From patchwork Fri Jan 5 16:00:08 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Github ODP bot X-Patchwork-Id: 123536 Delivered-To: patch@linaro.org Received: by 10.140.22.227 with SMTP id 90csp989024qgn; Fri, 5 Jan 2018 08:07:09 -0800 (PST) X-Google-Smtp-Source: ACJfBosbRLrC7ZuoOdueqQ7mFB/LZppKc/8HDwmKlTFndD1HC4PWH0OSe+Z8LNokV39iUrr2RwDZ X-Received: by 10.200.54.104 with SMTP id n37mr4426953qtb.271.1515168428917; Fri, 05 Jan 2018 08:07:08 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1515168428; cv=none; d=google.com; s=arc-20160816; b=tM5KqR7ae+qU4pEo+QKKnm9A1gGQwMeo5yG1MxbSJ2E+421ocfl7dl28Coepltuggl sVEFYCZSuTngwvSdt7aJiUXkYI+78dUwrfCF7pXzkH+l3GGpK88uZk/uiLw3IOZNHIY1 VsxNaM3IoVgqwGpAqqdn+O8w9zBK+g6jmzTWNEIMSTUOw8ce9ubor0r+B4cvqL377tWF 98QZyCJYHazXEilKLtNCYt2sY/d5MfH2BKRoxenlly3Y67FWT/TIVwFjC3+dXsMDbehJ Lf7432ValMLvRN4aobCHNxBkSdZTXM0n3k275piIdtdFNC3yE1aT0OGDnA/BI12btnbG +h0A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:github-pr-num :references:in-reply-to:message-id:date:to:from:delivered-to :arc-authentication-results; bh=QMdh70ZMbTQ6FkzDNAYJ/6DN4Q0hU1qpyEDlYhRhvmI=; b=ab6AEPKat6eNlpNrWVB42KVmwokrWTOzRVnYrXM3mxnwZGRKTGwU3zJchIjMv0YWgY x8ysDQYRebUWjVT5DVttljukh5AoUnFbpw7ioOfFiZaCl73tFsHcjNwUnsYByxuzw9QW u6NpXGKBn01EowmCxrwMtfDHJ+nPLprdl1GuoTnlMeF0G4NZGZ7ARh9TC3rIp4jum4p4 ofNzacprlA03ByUoUKJHmn8AsWkvsUnEgR3dOyddIcg0Vb6x8RmhLhjKHWVcixKrCjTR JaKMxhsDc2Xur0aL1ohba7ShHbbEFfkAdljpPjWWKAa5losIR5TpNWsJq1Sl0i28mRgW a+pg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.197.127.237 as permitted sender) smtp.mailfrom=lng-odp-bounces@lists.linaro.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=yandex.ru Return-Path: Received: from lists.linaro.org (ec2-54-197-127-237.compute-1.amazonaws.com. [54.197.127.237]) by mx.google.com with ESMTP id p2si5184723qkl.118.2018.01.05.08.07.08; Fri, 05 Jan 2018 08:07:08 -0800 (PST) Received-SPF: pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.197.127.237 as permitted sender) client-ip=54.197.127.237; Authentication-Results: mx.google.com; spf=pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.197.127.237 as permitted sender) smtp.mailfrom=lng-odp-bounces@lists.linaro.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=yandex.ru Received: by lists.linaro.org (Postfix, from userid 109) id 9BB07616F4; Fri, 5 Jan 2018 16:07:08 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on ip-10-142-244-252 X-Spam-Level: X-Spam-Status: No, score=-2.6 required=5.0 tests=BAYES_00,FREEMAIL_FROM, RCVD_IN_DNSWL_LOW,RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,URIBL_BLOCKED autolearn=disabled version=3.4.0 Received: from [127.0.0.1] (localhost [127.0.0.1]) by lists.linaro.org (Postfix) with ESMTP id 0AFBB61510; Fri, 5 Jan 2018 16:01:00 +0000 (UTC) X-Original-To: lng-odp@lists.linaro.org Delivered-To: lng-odp@lists.linaro.org Received: by lists.linaro.org (Postfix, from userid 109) id 9538B60C4C; Fri, 5 Jan 2018 16:00:29 +0000 (UTC) Received: from forward101j.mail.yandex.net (forward101j.mail.yandex.net [5.45.198.241]) by lists.linaro.org (Postfix) with ESMTPS id 935DA606A0 for ; Fri, 5 Jan 2018 16:00:19 +0000 (UTC) Received: from mxback7o.mail.yandex.net (mxback7o.mail.yandex.net [IPv6:2a02:6b8:0:1a2d::21]) by forward101j.mail.yandex.net (Yandex) with ESMTP id A150B12416A0 for ; Fri, 5 Jan 2018 19:00:12 +0300 (MSK) Received: from smtp2o.mail.yandex.net (smtp2o.mail.yandex.net [2a02:6b8:0:1a2d::26]) by mxback7o.mail.yandex.net (nwsmtp/Yandex) with ESMTP id yyZIbfLbKB-0Cd0CqXM; Fri, 05 Jan 2018 19:00:12 +0300 Received: by smtp2o.mail.yandex.net (nwsmtp/Yandex) with ESMTPSA id wLSnxxib0I-0BYa8mIj; Fri, 05 Jan 2018 19:00:11 +0300 (using TLSv1.2 with cipher ECDHE-RSA-AES128-SHA256 (128/128 bits)) (Client certificate not present) From: Github ODP bot To: lng-odp@lists.linaro.org Date: Fri, 5 Jan 2018 19:00:08 +0300 Message-Id: <1515168010-29173-2-git-send-email-odpbot@yandex.ru> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1515168010-29173-1-git-send-email-odpbot@yandex.ru> References: <1515168010-29173-1-git-send-email-odpbot@yandex.ru> Github-pr-num: 383 Subject: [lng-odp] [PATCH CATERPILLAR v1 1/3] linux-gen: mediated devices common code X-BeenThere: lng-odp@lists.linaro.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: "The OpenDataPlane \(ODP\) List" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: lng-odp-bounces@lists.linaro.org Sender: "lng-odp" From: Ilias Apalodimas Add APIs to create/destroy mediated devices, manipulate their regions and allocate/free DMA areas accessible by both ODP and HW. Signed-off-by: Ilias Apalodimas Signed-off-by: Mykyta Iziumtsev --- /** Email created from pull request 383 (MykytaI:caterpillar_mdev_cxgb4) ** https://github.com/Linaro/odp/pull/383 ** Patch: https://github.com/Linaro/odp/pull/383.patch ** Base sha: 4d17f8ae64aba0e6f24877be30f86ae5880cef7e ** Merge commit sha: 0e6e747c9f2997442be64a3cc70d7f117f29216d **/ platform/linux-dpdk/m4/configure.m4 | 1 + platform/linux-generic/Makefile.am | 3 + platform/linux-generic/m4/configure.m4 | 1 + platform/linux-generic/m4/odp_mdev.m4 | 34 ++ platform/linux-generic/pktio/mdev.c | 585 +++++++++++++++++++++++++++ platform/linux-generic/pktio/mdev.h | 61 +++ platform/linux-generic/pktio/uapi_net_mdev.h | 33 ++ 7 files changed, 718 insertions(+) create mode 100644 platform/linux-generic/m4/odp_mdev.m4 create mode 100644 platform/linux-generic/pktio/mdev.c create mode 100644 platform/linux-generic/pktio/mdev.h create mode 100644 platform/linux-generic/pktio/uapi_net_mdev.h diff --git a/platform/linux-dpdk/m4/configure.m4 b/platform/linux-dpdk/m4/configure.m4 index 9d299bf35..8c5e06707 100644 --- a/platform/linux-dpdk/m4/configure.m4 +++ b/platform/linux-dpdk/m4/configure.m4 @@ -80,6 +80,7 @@ AC_SUBST([ATOMIC_LIBS]) # linux-generic pktio at all. And DPDK has its own PCAP support anyway AM_CONDITIONAL([HAVE_PCAP], [false]) AM_CONDITIONAL([netmap_support], [false]) +AM_CONDITIONAL([mdev_support], [false]) AM_CONDITIONAL([PKTIO_DPDK], [false]) m4_include([platform/linux-dpdk/m4/odp_pthread.m4]) ODP_TIMER diff --git a/platform/linux-generic/Makefile.am b/platform/linux-generic/Makefile.am index 8813aea92..9a6a7cc15 100644 --- a/platform/linux-generic/Makefile.am +++ b/platform/linux-generic/Makefile.am @@ -181,6 +181,8 @@ noinst_HEADERS = \ pktio/common.h \ pktio/sysfs.h \ pktio/dpdk.h \ + pktio/mdev.h \ + pktio/uapi_net_mdev.h \ include/odp_pktio_ops_ipc.h \ include/odp_pktio_ops_loopback.h \ include/odp_pktio_ops_netmap.h \ @@ -255,6 +257,7 @@ __LIB__libodp_linux_la_SOURCES = \ pktio/sysfs.c \ pktio/tap.c \ pktio/ring.c \ + pktio/mdev.c \ pool/generic.c \ pool/subsystem.c \ odp_pkt_queue.c \ diff --git a/platform/linux-generic/m4/configure.m4 b/platform/linux-generic/m4/configure.m4 index fc29f9cb3..a0d65a319 100644 --- a/platform/linux-generic/m4/configure.m4 +++ b/platform/linux-generic/m4/configure.m4 @@ -14,6 +14,7 @@ m4_include([platform/linux-generic/m4/odp_modules.m4]) m4_include([platform/linux-generic/m4/odp_netmap.m4]) m4_include([platform/linux-generic/m4/odp_dpdk.m4]) m4_include([platform/linux-generic/m4/odp_schedule.m4]) +m4_include([platform/linux-generic/m4/odp_mdev.m4]) m4_include([platform/linux-generic/m4/performance.m4]) diff --git a/platform/linux-generic/m4/odp_mdev.m4 b/platform/linux-generic/m4/odp_mdev.m4 new file mode 100644 index 000000000..64f9e0969 --- /dev/null +++ b/platform/linux-generic/m4/odp_mdev.m4 @@ -0,0 +1,34 @@ +########################################################################## +# Enable mdev support +########################################################################## +mdev_support=no +AC_ARG_ENABLE([mdev_support], + [ --enable-mdev-support include mediated device drivers support], + [if test x$enableval = xyes; then + mdev_support=yes + fi]) + +########################################################################## +# Save and set temporary compilation flags +########################################################################## +OLD_CPPFLAGS=$CPPFLAGS +CPPFLAGS="$MDEV_CPPFLAGS $CPPFLAGS" + +########################################################################## +# Check for mdev availability +########################################################################## +if test x$mdev_support = xyes +then + AC_DEFINE([ODP_MDEV], [1], + [Define to 1 to enable mediated device drivers support]) + AC_SUBST([MDEV_CPPFLAGS]) +else + mdev_support=no +fi + +########################################################################## +# Restore old saved variables +########################################################################## +CPPFLAGS=$OLD_CPPFLAGS + +AM_CONDITIONAL([mdev_support], [test x$mdev_support = xyes ]) diff --git a/platform/linux-generic/pktio/mdev.c b/platform/linux-generic/pktio/mdev.c new file mode 100644 index 000000000..66b42185a --- /dev/null +++ b/platform/linux-generic/pktio/mdev.c @@ -0,0 +1,585 @@ +/* Copyright (c) 2018, Linaro Limited + * All rights reserved. + * + * SPDX-License-Identifier: BSD-3-Clause + */ + +#include "config.h" + +#ifdef ODP_MDEV + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include +#include + +#include +#include + +#include +#include + +/** + * returns a valid VFIO container + * fd must be closed by caller + */ +static int get_container(void) +{ + int ret; + int container; + /* Create a new container */ + container = open("/dev/vfio/vfio", O_RDWR); + + if (container < 0) + return container; + + ret = ioctl(container, VFIO_GET_API_VERSION); + if (ret != VFIO_API_VERSION) { + ODP_ERR("Unknown API version\n"); + goto out; + } + + if (!ioctl(container, VFIO_CHECK_EXTENSION, VFIO_TYPE1_IOMMU)) { + ODP_ERR("Doesn't support the IOMMU driver we want\n"); + goto out; + } + + return container; +out: + close(container); + container = -1; + return ret; +} + +/** + * returns a valid VFIO group + * fd must be close by caller + */ +static int get_group(int grp_id) +{ + char path[64]; + int ret; + int group; + struct vfio_group_status group_status = { + .argsz = sizeof(group_status) + }; + + snprintf(path, sizeof(path), "/dev/vfio/%d", grp_id); + group = open(path, O_RDWR); + if (group < 0) { + ODP_ERR("Failed to open %s, %d (%s)\n", + path, group, strerror(errno)); + return group; + } + + ret = ioctl(group, VFIO_GROUP_GET_STATUS, &group_status); + + if (ret < 0) { + ODP_ERR("ioctl(VFIO_GROUP_GET_STATUS) failed\n"); + goto out; + } + + /* Test the group is viable and available */ + if (!(group_status.flags & VFIO_GROUP_FLAGS_VIABLE)) { + ODP_ERR("Group is not viable\n"); + goto out; + } + + return group; +out: + close(group); + group = -1; + return ret; +} + +static void vfio_find_sparse_mmaps(struct vfio_info_cap_header *hdr, + struct vfio_region_info_cap_sparse_mmap + **sparse) +{ + *sparse = + odp_container_of(hdr, struct vfio_region_info_cap_sparse_mmap, + header); +} + +static struct vfio_info_cap_header * +vfio_get_region_info_cap(struct vfio_region_info *info, __u16 id) +{ + struct vfio_info_cap_header *hdr; + void *ptr = info; + + if (!(info->flags & VFIO_REGION_INFO_FLAG_CAPS)) + return NULL; + + for (hdr = + (struct vfio_info_cap_header *)((char *)ptr + info->cap_offset); + hdr != ptr; + hdr = (struct vfio_info_cap_header *)((char *)ptr + hdr->next)) { + if (hdr->id == id) + return hdr; + } + + return NULL; +} + +static struct vfio_info_cap_header * +vfio_get_cap_info(struct vfio_region_info *region_info, __u16 id) +{ + struct vfio_info_cap_header *caps = NULL; + + caps = vfio_get_region_info_cap(region_info, id); + + return caps; +} + +int vfio_get_region_sparse_mmaps(struct vfio_region_info *region_info, + struct vfio_region_info_cap_sparse_mmap + **sparse) +{ + struct vfio_info_cap_header *caps = NULL; + int ret = -ENOENT; + + if (region_info->flags & VFIO_REGION_INFO_FLAG_CAPS && + region_info->argsz > sizeof(*region_info)) { + caps = vfio_get_cap_info(region_info, + VFIO_REGION_INFO_CAP_SPARSE_MMAP); + if (!caps) + goto out; + vfio_find_sparse_mmaps(caps, sparse); + if (*sparse) { + for (uint32_t i = 0; i < (*sparse)->nr_areas; i++) + ODP_DBG("Sparse region: %d 0x%llx %llu\n", i, + (*sparse)->areas[i].offset, + (*sparse)->areas[i].size); + } + + ret = 0; + } + +out: + return ret; +} + +/** Match capability type + * returns 0 on succcess + */ +int vfio_get_region_cap_type(struct vfio_region_info *region_info, + mdev_region_class_t *class_info) +{ + struct vfio_info_cap_header *caps = NULL; + int ret = 0; + struct vfio_region_info_cap_type *cap_type; + + if (region_info->flags & VFIO_REGION_INFO_FLAG_CAPS && + region_info->argsz > sizeof(*region_info)) { + caps = vfio_get_cap_info(region_info, + VFIO_REGION_INFO_CAP_TYPE); + if (!caps) { + ret = -EINVAL; + goto out; + } + + cap_type = + odp_container_of(caps, struct vfio_region_info_cap_type, + header); + + class_info->type = cap_type->type; + class_info->subtype = cap_type->subtype; + } +out: + return ret; +} + +/** + * Get specific region info + */ +static struct vfio_region_info *vfio_get_region(mdev_device_t *mdev, + __u32 region) +{ + int ret; + struct vfio_region_info *region_info = NULL; + + ODP_DBG("Region:%d\n", region); + region_info = calloc(1, sizeof(*region_info)); + if (!region_info) + goto out; + + region_info->index = region; + region_info->argsz = sizeof(*region_info); + ret = ioctl(mdev->device, VFIO_DEVICE_GET_REGION_INFO, region_info); + if (ret < 0) { + ODP_ERR("Failed to get PCI region info\n"); + goto out; + } + + if (!region_info->size) { + ODP_DBG("region info %d is empty, skipping\n", region); + goto out; + } + + if (region_info->argsz > sizeof(*region_info)) { + struct vfio_region_info *tmp; + + tmp = realloc(region_info, region_info->argsz); + if (!tmp) + goto out; + + region_info = tmp; + + ODP_DBG("region info %d with extended capabilities size: %u\n", + region, region_info->argsz); + ret = ioctl(mdev->device, VFIO_DEVICE_GET_REGION_INFO, + region_info); + if (ret < 0 || !region_info->size) { + ODP_ERR("Failed to get PCI region info\n"); + goto out; + } + } + + return region_info; + +out: + if (region_info) + free(region_info); + return NULL; +} + +/** + * mmap a VFIO region + */ +void *mdev_region_mmap(mdev_device_t *mdev, uint64_t offset, uint64_t size) +{ + void *addr; + + /* Make sure we're page aligned */ + ODP_ASSERT(offset == ROUNDUP_ALIGN(offset, ODP_PAGE_SIZE)); + ODP_ASSERT(size == ROUNDUP_ALIGN(size, ODP_PAGE_SIZE)); + + if (mdev->mappings_count >= ARRAY_SIZE(mdev->mappings)) + return MAP_FAILED; + + addr = mmap(NULL, size, PROT_READ | PROT_WRITE, MAP_SHARED, + mdev->device, offset); + if (addr == MAP_FAILED) + return addr; + + mdev->mappings[mdev->mappings_count].addr = addr; + mdev->mappings[mdev->mappings_count].size = size; + mdev->mappings_count++; + + return addr; +} + +int mdev_dma_area_alloc(mdev_device_t *mdev, mdev_dma_area_t *dma_area) +{ + struct vfio_iommu_type1_dma_map req; + void *tmp; + + /* Make sure we're page aligned */ + if (dma_area->size != ROUNDUP_ALIGN(dma_area->size, ODP_PAGE_SIZE)) + return -EINVAL; + + memset(&req, 0, sizeof(req)); + req.argsz = sizeof(req); + + tmp = mmap(NULL, dma_area->size, PROT_READ | PROT_WRITE, + MAP_SHARED | MAP_ANONYMOUS | MAP_NORESERVE | MAP_FIXED, -1, + 0); + if (tmp == MAP_FAILED) { + ODP_ERR("mmap failed\n"); + return -EFAULT; + } + + dma_area->vaddr = (uint64_t)tmp; + + req.vaddr = dma_area->vaddr; + req.size = dma_area->size; + req.flags = VFIO_DMA_MAP_FLAG_READ | VFIO_DMA_MAP_FLAG_WRITE; + + if (ioctl(mdev->device, VFIO_IOMMU_MAP_DMA, &req) < 0) { + ODP_ERR("ioctl failed\n"); + return -EFAULT; + } + + dma_area->iova = req.iova; + + ODP_DBG("dma_area alloc: %llx@%llx -> %llx\n", dma_area->size, + dma_area->vaddr, dma_area->iova); + + return 0; +} + +int mdev_dma_area_free(mdev_device_t *mdev, mdev_dma_area_t *dma_area) +{ + struct vfio_iommu_type1_dma_unmap req; + + memset(&req, 0, sizeof(req)); + req.argsz = sizeof(req); + req.iova = dma_area->iova; + req.size = dma_area->size; + req.flags = VFIO_DMA_MAP_FLAG_READ | VFIO_DMA_MAP_FLAG_WRITE; + + if (ioctl(mdev->device, VFIO_IOMMU_UNMAP_DMA, &req) < 0) { + ODP_ERR("ioctl failed\n"); + return -EFAULT; + } + + if (munmap((void *)dma_area->vaddr, dma_area->size) < 0) { + ODP_ERR("munmap failed\n"); + return -EFAULT; + } + + ODP_DBG("dma_area free: %llx@%llx -> %llx\n", dma_area->size, + dma_area->vaddr, dma_area->iova); + + return 0; +} + +static char *mdev_basename(char *path) +{ + char *rpath; + + rpath = basename(path); + if (rpath) + return rpath; + + return NULL; +} + +static int mdev_readlink(const char *path, char *link, size_t linksz) +{ + ssize_t len; + + len = readlink(path, link, linksz - 1); + if (len != -1) { + link[len] = '\0'; + return 0; + } + return -1; +} + +/** + * returns group_id or -1 on fail and fills group_uuid + */ +static int mdev_sysfs_discover(const char *mod_name, const char *if_name, + char *uuid, size_t sz) +{ + int ret; + char *driver, *iommu_group; + char sysfs_path[2048], sysfs_link[2048]; + DIR *dir; + struct dirent *dp; + + /* Don't put / on the end of the path */ + snprintf(sysfs_path, sizeof(sysfs_path), + "/sys/class/net/%s/device/driver", if_name); + ret = mdev_readlink(sysfs_path, sysfs_link, sizeof(sysfs_link)); + if (ret) { + ODP_ERR("Can't locate sysfs driver path\n"); + return -1; + } + + driver = mdev_basename(sysfs_link); + if (!driver) { + ODP_ERR("Can't driver in sysfs\n"); + return -1; + } + + if (strcmp(driver, mod_name)) { + ODP_ERR("Invalid driver name\n"); + return -1; + } + + snprintf(sysfs_path, sizeof(sysfs_path), + "/sys/class/net/%s/device/mdev_supported_types/%s-netmdev/devices/", + if_name, driver); + + dir = opendir(sysfs_path); + if (!dir) + return -1; + /* FIXME only the last uuid will be returned now */ + while ((dp = readdir(dir))) { + if (!strcmp(dp->d_name, ".") || !strcmp(dp->d_name, "..")) + continue; + else + strncpy(uuid, dp->d_name, sz); + } + closedir(dir); + + if (!uuid[0]) + return -1; + + snprintf(sysfs_path, sizeof(sysfs_path), + "/sys/bus/mdev/devices/%s/iommu_group", uuid); + ret = mdev_readlink(sysfs_path, sysfs_link, sizeof(sysfs_link)); + if (ret) { + ODP_ERR("Can't locate IOMMU sysfs path\n"); + return -1; + } + + iommu_group = mdev_basename(sysfs_link); + if (!iommu_group) { + ODP_ERR("Can't locate iommu group in sysfs\n"); + return -1; + } + ret = atoi(iommu_group); + + return ret; +} + +/** + * Initialize VFIO variables. + * set IOMMU and get device regions + */ +static int vfio_init_dev(int grp, int container, + struct vfio_group_status *grp_status, + struct vfio_iommu_type1_info *iommu_info, + struct vfio_device_info *dev_info, char *grp_uuid) +{ + int device = -1; + int ret; + + ret = ioctl(container, VFIO_CHECK_EXTENSION, VFIO_TYPE1_IOMMU); + if (ret < 0) { + ODP_ERR("Doesn't support the IOMMU driver we want\n"); + goto out; + } + + /* Test the group is viable and available */ + ret = ioctl(grp, VFIO_GROUP_GET_STATUS, grp_status); + if (ret < 0 || !(grp_status->flags & VFIO_GROUP_FLAGS_VIABLE)) { + ODP_ERR("Can't get status\n"); + goto out; + } + + ret = ioctl(grp, VFIO_GROUP_SET_CONTAINER, &container); + if (ret < 0) { + ODP_ERR("Failed to set container\n"); + goto out; + } + + ret = ioctl(container, VFIO_SET_IOMMU, VFIO_TYPE1_IOMMU); + if (ret < 0) { + ODP_ERR("Failed to set IOMMU\n"); + goto out; + } + + ret = ioctl(container, VFIO_IOMMU_GET_INFO, iommu_info); + if (ret < 0) { + ODP_ERR("Failed to get IOMMU info\n"); + goto out; + } + + ODP_DBG("iova_pgsizes bitmask=0x%llx\n", iommu_info->iova_pgsizes); + /* Get a file descriptor for the device */ + device = ioctl(grp, VFIO_GROUP_GET_DEVICE_FD, grp_uuid); + if (device < 0) { + ODP_ERR("Failed to get device FD\n"); + goto out; + } + + /* Test and setup the device */ + ret = ioctl(device, VFIO_DEVICE_GET_INFO, dev_info); + if (ret < 0) { + ODP_ERR("Failed to get device info\n"); + goto out; + } + + ODP_DBG("Device %d Regions: %d, irqs:%d\n", device, + dev_info->num_regions, dev_info->num_irqs); + +out: + return device; +} + +int mdev_device_create(mdev_device_t *mdev, const char *mod_name, + const char *if_name, + mdev_region_info_cb_t region_info_cb) +{ + struct vfio_group_status group_status = { + .argsz = sizeof(group_status) + }; + struct vfio_iommu_type1_info iommu_info = { + .argsz = sizeof(iommu_info) + }; + struct vfio_device_info device_info = { + .argsz = sizeof(device_info) + }; + int ret; + + memset(mdev, 0, sizeof(*mdev)); + mdev->container = -1; + mdev->group = -1; + + strncpy(mdev->if_name, if_name, sizeof(mdev->if_name) - 1); + + mdev->group_id = + mdev_sysfs_discover(mod_name, mdev->if_name, mdev->group_uuid, + sizeof(mdev->group_uuid)); + if (mdev->group_id < 0) + goto fail; + + mdev->container = get_container(); + if (mdev->container < 0) + goto fail; + + mdev->group = get_group(mdev->group_id); + if (mdev->group < 0) + goto fail; + + mdev->device = + vfio_init_dev(mdev->group, mdev->container, &group_status, + &iommu_info, &device_info, mdev->group_uuid); + if (mdev->device < 0) + goto fail; + + ret = -EINVAL; + for (uint32_t region = 0; region < device_info.num_regions; region++) { + struct vfio_region_info *region_info; + + region_info = vfio_get_region(mdev, region); + if (!region_info) + continue; + + ret = region_info_cb(mdev, region_info); + if (ret < 0) { + ODP_ERR("Region info cb fail on region_info[%u]\n", + region); + return -1; + } + + free(region_info); + region_info = NULL; + ret = 0; + } + + return ret; + +fail: + return -1; +} + +void mdev_device_destroy(mdev_device_t *mdev) +{ + if (mdev->group != -1) + close(mdev->group); + if (mdev->container != -1) + close(mdev->container); + + for (uint16_t i = 0; i < mdev->mappings_count; i++) + munmap(mdev->mappings[i].addr, mdev->mappings[i].size); +} + +#endif /* ODP_MDEV */ diff --git a/platform/linux-generic/pktio/mdev.h b/platform/linux-generic/pktio/mdev.h new file mode 100644 index 000000000..5249a6d05 --- /dev/null +++ b/platform/linux-generic/pktio/mdev.h @@ -0,0 +1,61 @@ +/* Copyright (c) 2018, Linaro Limited + * All rights reserved. + * + * SPDX-License-Identifier: BSD-3-Clause + */ + +#ifndef ODP_PKTIO_MDEV_H_ +#define ODP_PKTIO_MDEV_H_ + +#include +#include + +#define NET_MDEV_PREFIX "mdev:" + +typedef struct { + uint16_t type; + uint16_t subtype; +} mdev_region_class_t; + +typedef struct { + uint64_t vaddr; + uint64_t iova; + uint64_t size; +} mdev_dma_area_t; + +typedef struct { + char if_name[IF_NAMESIZE]; /**< Interface name */ + + int container; + int group; + int device; + + int group_id; + char group_uuid[64]; + + struct { + uint8_t *addr; + size_t size; + } mappings[256]; + uint16_t mappings_count; +} mdev_device_t; + +typedef int (*mdev_region_info_cb_t)(mdev_device_t *, + struct vfio_region_info *); + +int mdev_device_create(mdev_device_t *mdev, const char *mod_name, + const char *if_name, mdev_region_info_cb_t cb); +void mdev_device_destroy(mdev_device_t *mdev); + +void *mdev_region_mmap(mdev_device_t *mdev, uint64_t offset, uint64_t size); + +int vfio_get_region_cap_type(struct vfio_region_info *region_info, + mdev_region_class_t *type_info); +int vfio_get_region_sparse_mmaps(struct vfio_region_info *region_info, + struct vfio_region_info_cap_sparse_mmap + **sparse); + +int mdev_dma_area_alloc(mdev_device_t *mdev, mdev_dma_area_t *dma_area); +int mdev_dma_area_free(mdev_device_t *mdev, mdev_dma_area_t *dma_area); + +#endif diff --git a/platform/linux-generic/pktio/uapi_net_mdev.h b/platform/linux-generic/pktio/uapi_net_mdev.h new file mode 100644 index 000000000..b0445fc68 --- /dev/null +++ b/platform/linux-generic/pktio/uapi_net_mdev.h @@ -0,0 +1,33 @@ +/* Copyright (c) 2018, Linaro Limited + * All rights reserved. + * + * SPDX-License-Identifier: BSD-3-Clause + */ + +#ifndef _UAPI__LINUX_NET_MDEV_H +#define _UAPI__LINUX_NET_MDEV_H + +#include +#include + +enum net_mdev_types { + VFIO_NET_MDEV_SHADOW, + VFIO_NET_DESCRIPTORS, + VFIO_NET_MMIO, +}; + +enum net_mdev_subtypes { + VFIO_NET_MDEV_STATS, + VFIO_NET_MDEV_RX, + VFIO_NET_MDEV_TX, + VFIO_NET_MDEV_BARS, +}; + +enum vfio_net_mdev_regions { + VFIO_NET_MDEV_SHADOW_REGION_INDEX, + VFIO_NET_MDEV_RX_REGION_INDEX, + VFIO_NET_MDEV_TX_REGION_INDEX, + VFIO_NET_MDEV_NUM_REGIONS, +}; + +#endif /* _UAPI__LINUX_NET_MDEV_H */ From patchwork Fri Jan 5 16:00:09 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Github ODP bot X-Patchwork-Id: 123534 Delivered-To: patch@linaro.org Received: by 10.140.22.227 with SMTP id 90csp980985qgn; Fri, 5 Jan 2018 08:01:15 -0800 (PST) X-Google-Smtp-Source: ACJfBovN+5f266FIAOt7cNc4EXc9uT1manitrnQIYFc/CckVLY9m1tmNRdntD7ffYbqipHoXfHC9 X-Received: by 10.55.116.68 with SMTP id p65mr4832207qkc.140.1515168075397; Fri, 05 Jan 2018 08:01:15 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1515168075; cv=none; d=google.com; s=arc-20160816; b=lQI+LvC5ThjoxzKtE/QlLZ05PmJZk3e8A2UCJ1YkJrzR1SX4gL7EGFDeyRx09xlHhs srEYshv09tN2SN7/8WFpvEOyFyVhSfT/if48RgR1kKh0agEHoLjuStLFEgyvxJKQZg9z gWxR5etCHtZ3H8dngIiiM4U9a3AC4qAi3FOB6w6Yw82liTpJHUMhib+p/Lb6Jtwj3W4X JE94Z8s2jl2JiWSxKBaoqqN52QFWedcqEEZDcFgtp2eCyCQ0p/BGyCq4VwDrb0M3cBDw kjGICj1tOZRmz/0mXN6e+QQb90bTZ6UOzPq2mFZ6kJp22Mw/BicxMqUtjNeVYerXWWR2 5oYQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:github-pr-num :references:in-reply-to:message-id:date:to:from:delivered-to :arc-authentication-results; bh=ogEEBttZJxeAEcq2uVCxsSfjvUaxl+6oomndj9nC6jY=; b=cR3dIIaTOHxH3P2go5kg13AFbj0JFHe1blhUCMuIKj2Ry1rxu8H8VB/zrvYhBdkMbX wXFXzVcY+NGPNIN7h9Zl3Bql6rpjfeMPlB3QkYEW8Vy082huU6hm3h64kZfSQ7bnplDl xCLn0h7pdSvbFWTUhgFKKO+/oQYZg/8C5vQa7nnbSgu9b7+3C4RweUCqQJfbmnZvCg2y c0R8T4Thv0Q2+kvqMOI4DO5W0izTFrwbp4lQ5XXDRwPNGUcxqKwg12RCT9e6t22WVFlo DD9/89Rfv3p7VRAvoBQuJOyoI2nonMj76EtYIVM0SkBA5MAUAIBrHHBaFZX7Aba8x96S aGbw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.197.127.237 as permitted sender) smtp.mailfrom=lng-odp-bounces@lists.linaro.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=yandex.ru Return-Path: Received: from lists.linaro.org (ec2-54-197-127-237.compute-1.amazonaws.com. [54.197.127.237]) by mx.google.com with ESMTP id x35si150168qte.182.2018.01.05.08.01.15; Fri, 05 Jan 2018 08:01:15 -0800 (PST) Received-SPF: pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.197.127.237 as permitted sender) client-ip=54.197.127.237; Authentication-Results: mx.google.com; spf=pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.197.127.237 as permitted sender) smtp.mailfrom=lng-odp-bounces@lists.linaro.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=yandex.ru Received: by lists.linaro.org (Postfix, from userid 109) id 1259160B52; Fri, 5 Jan 2018 16:01:15 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on ip-10-142-244-252 X-Spam-Level: X-Spam-Status: No, score=-2.6 required=5.0 tests=BAYES_00,FREEMAIL_FROM, RCVD_IN_DNSWL_LOW, RCVD_IN_MSPIKE_H2, URIBL_BLOCKED autolearn=disabled version=3.4.0 Received: from [127.0.0.1] (localhost [127.0.0.1]) by lists.linaro.org (Postfix) with ESMTP id B9CA160BF0; Fri, 5 Jan 2018 16:00:30 +0000 (UTC) X-Original-To: lng-odp@lists.linaro.org Delivered-To: lng-odp@lists.linaro.org Received: by lists.linaro.org (Postfix, from userid 109) id 93DCE60AEF; Fri, 5 Jan 2018 16:00:17 +0000 (UTC) Received: from forward100p.mail.yandex.net (forward100p.mail.yandex.net [77.88.28.100]) by lists.linaro.org (Postfix) with ESMTPS id 29773606A0 for ; Fri, 5 Jan 2018 16:00:15 +0000 (UTC) Received: from mxback6j.mail.yandex.net (mxback6j.mail.yandex.net [IPv6:2a02:6b8:0:1619::10f]) by forward100p.mail.yandex.net (Yandex) with ESMTP id 8DF245102BD5 for ; Fri, 5 Jan 2018 19:00:13 +0300 (MSK) Received: from smtp2o.mail.yandex.net (smtp2o.mail.yandex.net [2a02:6b8:0:1a2d::26]) by mxback6j.mail.yandex.net (nwsmtp/Yandex) with ESMTP id WJ2PNrg0RQ-0DiG1Xxx; Fri, 05 Jan 2018 19:00:13 +0300 Received: by smtp2o.mail.yandex.net (nwsmtp/Yandex) with ESMTPSA id wLSnxxib0I-0CYW3tZR; Fri, 05 Jan 2018 19:00:12 +0300 (using TLSv1.2 with cipher ECDHE-RSA-AES128-SHA256 (128/128 bits)) (Client certificate not present) From: Github ODP bot To: lng-odp@lists.linaro.org Date: Fri, 5 Jan 2018 19:00:09 +0300 Message-Id: <1515168010-29173-3-git-send-email-odpbot@yandex.ru> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1515168010-29173-1-git-send-email-odpbot@yandex.ru> References: <1515168010-29173-1-git-send-email-odpbot@yandex.ru> Github-pr-num: 383 Subject: [lng-odp] [PATCH CATERPILLAR v1 2/3] linux-gen: add i40e mediated device driver X-BeenThere: lng-odp@lists.linaro.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: "The OpenDataPlane \(ODP\) List" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: lng-odp-bounces@lists.linaro.org Sender: "lng-odp" From: Mykyta Iziumtsev Signed-off-by: Mykyta Iziumtsev --- /** Email created from pull request 383 (MykytaI:caterpillar_mdev_cxgb4) ** https://github.com/Linaro/odp/pull/383 ** Patch: https://github.com/Linaro/odp/pull/383.patch ** Base sha: 4d17f8ae64aba0e6f24877be30f86ae5880cef7e ** Merge commit sha: 0e6e747c9f2997442be64a3cc70d7f117f29216d **/ platform/linux-generic/Makefile.am | 1 + platform/linux-generic/pktio/mdev/i40e.c | 641 +++++++++++++++++++++++++++++++ platform/linux-generic/pktio/subsystem.c | 6 + 3 files changed, 648 insertions(+) create mode 100644 platform/linux-generic/pktio/mdev/i40e.c diff --git a/platform/linux-generic/Makefile.am b/platform/linux-generic/Makefile.am index 9a6a7cc15..85f902115 100644 --- a/platform/linux-generic/Makefile.am +++ b/platform/linux-generic/Makefile.am @@ -258,6 +258,7 @@ __LIB__libodp_linux_la_SOURCES = \ pktio/tap.c \ pktio/ring.c \ pktio/mdev.c \ + pktio/mdev/i40e.c \ pool/generic.c \ pool/subsystem.c \ odp_pkt_queue.c \ diff --git a/platform/linux-generic/pktio/mdev/i40e.c b/platform/linux-generic/pktio/mdev/i40e.c new file mode 100644 index 000000000..5797e9a5e --- /dev/null +++ b/platform/linux-generic/pktio/mdev/i40e.c @@ -0,0 +1,641 @@ +/* Copyright (c) 2018, Linaro Limited + * All rights reserved. + * + * SPDX-License-Identifier: BSD-3-Clause + */ + +#include "config.h" + +#ifdef ODP_MDEV + +#include + +#include +#include +#include +#include +#include +#include + +#include +#include +#include + +#include +#include + +#include + +#include + +#include +#include +#include +#include +#include + +#define MODULE_NAME "i40e" + +#define I40E_TX_BUF_SIZE 2048U +#define I40E_RX_BUF_SIZE 2048U + +#define I40E_TX_PACKET_LEN_MIN 17 + +/* RX queue definitions */ +#define I40E_RX_QUEUE_NUM_MAX 32 + +/** RX descriptor */ +typedef struct { + odp_u64le_t addr; +#define I40E_RXD_QW1_LEN_S 38 +#define I40E_RXD_QW1_LEN_M (0x3FFFULL << I40E_RXD_QW1_LEN_S) + odp_u64le_t status_error_len; + odp_u64le_t reserved[2]; +} i40e_rx_desc_t; + +/** RX queue data */ +typedef struct { + i40e_rx_desc_t *rx_descs; /**< RX queue base */ + odp_u32le_t *doorbell; /**< RX queue doorbell */ + + uint16_t rx_queue_len; /**< Number of RX desc entries */ + uint16_t cidx; /**< Next RX desc to handle */ + + mdev_dma_area_t rx_data; /**< RX packet payload area */ + + odp_ticketlock_t lock; /**< RX queue lock */ +} i40e_rx_queue_t /* ODP_ALIGNED_CACHE */; + +/* TX queue definitions */ +#define I40E_TX_QUEUE_NUM_MAX 32 + +typedef struct { + odp_u64le_t addr; + +#define I40E_TXD_DTYPE_DATA 0UL + +#define I40E_TXD_CMD_EOP 0x0001UL +#define I40E_TXD_CMD_RS 0x0002UL +#define I40E_TXD_CMD_ICRC 0x0004UL + +#define I40E_TXD_QW1_CMD_S 4 +#define I40E_TXD_QW1_L2TAG1_S 48 +#define I40E_TXD_QW1_OFFSET_S 16 +#define I40E_TXD_QW1_LEN_S 34 + odp_u64le_t cmd_type_offset_len; +} i40e_tx_desc_t; + +/** TX queue data */ +typedef struct { + i40e_tx_desc_t *tx_descs; /**< TX queue base */ + odp_u32le_t *doorbell; /**< TX queue doorbell */ + + uint16_t tx_queue_len; /**< Number of TX desc entries */ + uint16_t pidx; /**< Next TX desc to insert */ + + odp_u32le_t *cidx; /**< Last TX desc processed by HW */ + + mdev_dma_area_t tx_data; /**< TX packet payload area */ + + odp_ticketlock_t lock; /**< TX queue lock */ +} i40e_tx_queue_t /* ODP_ALIGNED_CACHE */; + +/** Packet socket using mediated i40e device */ +typedef struct { + /** RX queue hot data */ + i40e_rx_queue_t rx_queues[I40E_RX_QUEUE_NUM_MAX]; + + /** TX queue hot data */ + i40e_tx_queue_t tx_queues[I40E_TX_QUEUE_NUM_MAX]; + + odp_pool_t pool; /**< pool to alloc packets from */ + + odp_bool_t lockless_rx; /**< no locking for RX */ + odp_bool_t lockless_tx; /**< no locking for TX */ + + odp_pktio_capability_t capa; /**< interface capabilities */ + + uint8_t *mmio; /**< MMIO region */ + + int sockfd; /**< control socket */ + + mdev_device_t mdev; /**< Common mdev data */ +} pktio_ops_i40e_data_t; + +static void i40e_rx_refill(i40e_rx_queue_t *rxq, uint16_t from, uint16_t num); +static void i40e_wait_link_up(pktio_entry_t *pktio_entry); +static int i40e_close(pktio_entry_t *pktio_entry); + +static int i40e_mmio_register(pktio_ops_i40e_data_t *pkt_i40e, + uint64_t offset, uint64_t size) +{ + ODP_ASSERT(pkt_i40e->mmio == NULL); + + pkt_i40e->mmio = mdev_region_mmap(&pkt_i40e->mdev, offset, size); + if (pkt_i40e->mmio == MAP_FAILED) { + ODP_ERR("Cannot mmap MMIO\n"); + return -1; + } + + ODP_DBG("Register MMIO region: 0x%llx@%016llx\n", size, offset); + + return 0; +} + +static int i40e_rx_queue_register(pktio_ops_i40e_data_t *pkt_i40e, + uint64_t offset, uint64_t size) +{ + uint16_t rxq_idx = pkt_i40e->capa.max_input_queues++; + i40e_rx_queue_t *rxq = &pkt_i40e->rx_queues[rxq_idx]; + uint64_t doorbell_offset; + struct ethtool_ringparam ering; + int ret; + + ODP_ASSERT(rxq_idx < ARRAY_SIZE(pkt_i40e->rx_queues)); + + odp_ticketlock_init(&rxq->lock); + + ret = ethtool_ringparam_get_fd(pkt_i40e->sockfd, + pkt_i40e->mdev.if_name, &ering); + if (ret) { + ODP_ERR("Cannot get queue length\n"); + return -1; + } + rxq->rx_queue_len = ering.rx_pending; + + ret = sysfs_attr_u64_get(&doorbell_offset, + "/sys/class/net/%s" + "/queues/rx-%u/i40e/doorbell_offset", + pkt_i40e->mdev.if_name, rxq_idx); + if (ret) { + ODP_ERR("Cannot get %s rx-%u doorbell_offset\n", + pkt_i40e->mdev.if_name, rxq_idx); + return -1; + } + rxq->doorbell = (odp_u32le_t *)(pkt_i40e->mmio + doorbell_offset); + + ODP_ASSERT(rxq->rx_queue_len * sizeof(*rxq->rx_descs) <= size); + + rxq->rx_descs = mdev_region_mmap(&pkt_i40e->mdev, offset, size); + if (rxq->rx_descs == MAP_FAILED) { + ODP_ERR("Cannot mmap RX queue\n"); + return -1; + } + + rxq->rx_data.size = rxq->rx_queue_len * I40E_RX_BUF_SIZE; + ret = mdev_dma_area_alloc(&pkt_i40e->mdev, &rxq->rx_data); + if (ret) { + ODP_ERR("Cannot allocate RX queue DMA area\n"); + return -1; + } + + /* Need 1 desc gap to keep tail from touching head */ + i40e_rx_refill(rxq, 0, rxq->rx_queue_len - 1); + + ODP_DBG("Register RX queue region: 0x%llx@%016llx\n", size, offset); + ODP_DBG(" RX descriptors: %u\n", rxq->rx_queue_len); + + return 0; +} + +static int i40e_tx_queue_register(pktio_ops_i40e_data_t *pkt_i40e, + uint64_t offset, uint64_t size) +{ + uint16_t txq_idx = pkt_i40e->capa.max_output_queues++; + i40e_tx_queue_t *txq = &pkt_i40e->tx_queues[txq_idx]; + uint64_t doorbell_offset; + struct ethtool_ringparam ering; + int ret; + + ODP_ASSERT(txq_idx < ARRAY_SIZE(pkt_i40e->tx_queues)); + + odp_ticketlock_init(&txq->lock); + + ret = ethtool_ringparam_get_fd(pkt_i40e->sockfd, + pkt_i40e->mdev.if_name, &ering); + if (ret) { + ODP_ERR("Cannot get queue length\n"); + return -1; + } + txq->tx_queue_len = ering.tx_pending; + + ret = sysfs_attr_u64_get(&doorbell_offset, + "/sys/class/net/%s" + "/queues/tx-%u/i40e/doorbell_offset", + pkt_i40e->mdev.if_name, txq_idx); + if (ret) { + ODP_ERR("Cannot get %s tx-%u doorbell_offset\n", + pkt_i40e->mdev.if_name, txq_idx); + return -1; + } + txq->doorbell = (odp_u32le_t *)(pkt_i40e->mmio + doorbell_offset); + + ODP_ASSERT(txq->tx_queue_len * sizeof(*txq->tx_descs) + + sizeof(*txq->cidx) <= size); + + txq->tx_descs = mdev_region_mmap(&pkt_i40e->mdev, offset, size); + if (txq->tx_descs == MAP_FAILED) { + ODP_ERR("Cannot mmap TX queue\n"); + return -1; + } + + txq->cidx = (odp_u32le_t *)(txq->tx_descs + txq->tx_queue_len); + + txq->tx_data.size = txq->tx_queue_len * I40E_TX_BUF_SIZE; + ret = mdev_dma_area_alloc(&pkt_i40e->mdev, &txq->tx_data); + if (ret) { + ODP_ERR("Cannot allocate TX queue DMA area\n"); + return -1; + } + + ODP_DBG("Register TX queue region: 0x%llx@%016llx\n", size, offset); + ODP_DBG(" TX descriptors: %u\n", txq->tx_queue_len); + + return 0; +} + +static int i40e_region_info_cb(mdev_device_t *mdev, + struct vfio_region_info *region_info) +{ + pktio_ops_i40e_data_t *pkt_i40e = + odp_container_of(mdev, pktio_ops_i40e_data_t, mdev); + mdev_region_class_t class_info; + + if (vfio_get_region_cap_type(region_info, &class_info) < 0) { + ODP_ERR("Cannot find class_info in region %u\n", + region_info->index); + return -1; + } + + switch (class_info.type) { + case VFIO_NET_MMIO: + return i40e_mmio_register(pkt_i40e, + region_info->offset, + region_info->size); + + case VFIO_NET_DESCRIPTORS: + if (class_info.subtype == VFIO_NET_MDEV_RX) + return i40e_rx_queue_register(pkt_i40e, + region_info->offset, + region_info->size); + if (class_info.subtype == VFIO_NET_MDEV_TX) + return i40e_tx_queue_register(pkt_i40e, + region_info->offset, + region_info->size); + /* fallthrough */ + + default: + ODP_ERR("Unexpected region %u (class %u:%u)\n", + region_info->index, class_info.type, + class_info.subtype); + return -1; + } +} + +static int i40e_open(odp_pktio_t id ODP_UNUSED, + pktio_entry_t *pktio_entry, + const char *resource, odp_pool_t pool) +{ + pktio_ops_i40e_data_t *pkt_i40e; + int ret; + + ODP_ASSERT(pool != ODP_POOL_INVALID); + + if (strncmp(resource, NET_MDEV_PREFIX, strlen(NET_MDEV_PREFIX))) + return -1; + + ODP_DBG("%s: probing resource %s\n", MODULE_NAME, resource); + + pkt_i40e = ODP_OPS_DATA_ALLOC(sizeof(*pkt_i40e)); + if (odp_unlikely(pkt_i40e == NULL)) { + ODP_ERR("Failed to allocate pktio_ops_i40e_data_t struct"); + return -1; + } + pktio_entry->s.ops_data = pkt_i40e; + + memset(pkt_i40e, 0, sizeof(*pkt_i40e)); + + pkt_i40e->sockfd = socket(AF_INET, SOCK_DGRAM, 0); + if (pkt_i40e->sockfd == -1) { + ODP_ERR("Cannot get device control socket\n"); + goto out; + } + + ret = + mdev_device_create(&pkt_i40e->mdev, MODULE_NAME, + resource + strlen(NET_MDEV_PREFIX), + i40e_region_info_cb); + if (ret) + goto out; + + pkt_i40e->pool = pool; + + i40e_wait_link_up(pktio_entry); + + ODP_DBG("%s: open %s is successful\n", MODULE_NAME, + pkt_i40e->mdev.if_name); + + return 0; + +out: + i40e_close(pktio_entry); + return -1; +} + +static int i40e_close(pktio_entry_t *pktio_entry) +{ + pktio_ops_i40e_data_t *pkt_i40e = pktio_entry->s.ops_data; + + ODP_DBG("%s: close %s\n", MODULE_NAME, pkt_i40e->mdev.if_name); + + mdev_device_destroy(&pkt_i40e->mdev); + + for (uint16_t i = 0; i < pkt_i40e->capa.max_input_queues; i++) { + i40e_rx_queue_t *rxq = &pkt_i40e->rx_queues[i]; + + if (rxq->rx_data.size) + mdev_dma_area_free(&pkt_i40e->mdev, &rxq->rx_data); + } + + for (uint16_t i = 0; i < pkt_i40e->capa.max_output_queues; i++) { + i40e_tx_queue_t *txq = &pkt_i40e->tx_queues[i]; + + if (txq->tx_data.size) + mdev_dma_area_free(&pkt_i40e->mdev, &txq->tx_data); + } + + if (pkt_i40e->sockfd != -1) + close(pkt_i40e->sockfd); + + ODP_OPS_DATA_FREE(pkt_i40e); + + return 0; +} + +static void i40e_rx_refill(i40e_rx_queue_t *rxq, uint16_t from, uint16_t num) +{ + uint16_t i = from; + + while (num) { + uint64_t iova = rxq->rx_data.iova + i * I40E_RX_BUF_SIZE; + i40e_rx_desc_t *rxd = &rxq->rx_descs[i]; + + rxd->addr = odp_cpu_to_le_64(iova); + rxd->status_error_len = odp_cpu_to_le_64(0); + + i++; + if (i >= rxq->rx_queue_len) + i = 0; + + num--; + } + + /* Ring the doorbell */ + odpdrv_mmio_u32le_write(i, rxq->doorbell); +} + +static int i40e_recv(pktio_entry_t *pktio_entry, int rxq_idx, + odp_packet_t pkt_table[], int num) +{ + pktio_ops_i40e_data_t *pkt_i40e = pktio_entry->s.ops_data; + i40e_rx_queue_t *rxq = &pkt_i40e->rx_queues[rxq_idx]; + uint16_t refill_from; + int rx_pkts = 0; + + if (!pkt_i40e->lockless_rx) + odp_ticketlock_lock(&rxq->lock); + + /* Keep track of the start point to refill RX queue */ + refill_from = rxq->cidx ? rxq->cidx - 1 : rxq->rx_queue_len - 1; + + while (rx_pkts < num) { + volatile i40e_rx_desc_t *rxd = &rxq->rx_descs[rxq->cidx]; + odp_packet_hdr_t *pkt_hdr; + odp_packet_t pkt; + uint16_t pkt_len; + uint64_t status_error_len; + int ret; + + status_error_len = odp_le_to_cpu_64(rxd->status_error_len); + if (!status_error_len) + break; + + pkt_len = (status_error_len & I40E_RXD_QW1_LEN_M) >> + I40E_RXD_QW1_LEN_S; + + pkt = odp_packet_alloc(pkt_i40e->pool, pkt_len); + if (odp_unlikely(pkt == ODP_PACKET_INVALID)) + break; + + pkt_hdr = odp_packet_hdr(pkt); + + ret = odp_packet_copy_from_mem(pkt, 0, pkt_len, + (uint8_t *)rxq->rx_data.vaddr + + rxq->cidx * I40E_RX_BUF_SIZE); + if (odp_unlikely(ret)) { + odp_packet_free(pkt); + break; + } + + pkt_hdr->input = pktio_entry->s.handle; + + rxq->cidx++; + if (odp_unlikely(rxq->cidx >= rxq->rx_queue_len)) + rxq->cidx = 0; + + pkt_table[rx_pkts] = pkt; + rx_pkts++; + } + + if (rx_pkts) + i40e_rx_refill(rxq, refill_from, rx_pkts); + + if (!pkt_i40e->lockless_rx) + odp_ticketlock_unlock(&rxq->lock); + + return rx_pkts; +} + +/** + * Helper function to build descriptor 2nd quad-word. + */ +static inline uint64_t txd_ctol(uint32_t cmd, uint32_t tag, uint32_t offset, + uint32_t len) +{ + return I40E_TXD_DTYPE_DATA | + ((uint64_t)cmd << I40E_TXD_QW1_CMD_S) | + ((uint64_t)tag << I40E_TXD_QW1_L2TAG1_S) | + ((uint64_t)offset << I40E_TXD_QW1_OFFSET_S) | + ((uint64_t)len << I40E_TXD_QW1_LEN_S); +} + +static int i40e_send(pktio_entry_t *pktio_entry, int txq_idx, + const odp_packet_t pkt_table[], int num) +{ + pktio_ops_i40e_data_t *pkt_i40e = pktio_entry->s.ops_data; + i40e_tx_queue_t *txq = &pkt_i40e->tx_queues[txq_idx]; + uint16_t budget, tx_txds = 0; + int tx_pkts = 0; + + if (!pkt_i40e->lockless_tx) + odp_ticketlock_lock(&txq->lock); + + /* Determine how many packets will fit in TX queue */ + budget = txq->tx_queue_len - 1; + budget -= txq->pidx; + budget += odp_le_to_cpu_32(*txq->cidx); + budget &= txq->tx_queue_len - 1; + + while (tx_txds < budget && tx_pkts < num) { + uint16_t pkt_len = _odp_packet_len(pkt_table[tx_pkts]); + uint32_t offset = txq->pidx * I40E_TX_BUF_SIZE; + + i40e_tx_desc_t *txd = &txq->tx_descs[txq->pidx]; + + uint32_t txd_cmd = I40E_TXD_CMD_ICRC | I40E_TXD_CMD_EOP; + + /* Skip undersized packets silently */ + if (odp_unlikely(pkt_len < I40E_TX_PACKET_LEN_MIN)) { + tx_pkts++; + continue; + } + + /* Skip oversized packets silently */ + if (odp_unlikely(pkt_len > I40E_TX_BUF_SIZE)) { + tx_pkts++; + continue; + } + + /* Request CIDX update from firmware from time to time */ + if (!(txq->pidx & ((txq->tx_queue_len >> 2) - 1))) + txd_cmd |= I40E_TXD_CMD_RS; + + odp_packet_copy_to_mem(pkt_table[tx_pkts], 0, pkt_len, + (uint8_t *)txq->tx_data.vaddr + offset); + + txd->addr = odp_cpu_to_le_64(txq->tx_data.iova + offset); + txd->cmd_type_offset_len = + odp_cpu_to_le_64(txd_ctol(txd_cmd, 0, 0, pkt_len)); + + txq->pidx++; + if (odp_unlikely(txq->pidx >= txq->tx_queue_len)) + txq->pidx -= txq->tx_queue_len; + + tx_txds++; + tx_pkts++; + } + + /* Ring the doorbell */ + if (tx_pkts) + odpdrv_mmio_u32le_write(txq->pidx, txq->doorbell); + + if (!pkt_i40e->lockless_tx) + odp_ticketlock_unlock(&txq->lock); + + odp_packet_free_multi(pkt_table, tx_pkts); + + return tx_pkts; +} + +static int i40e_link_status(pktio_entry_t *pktio_entry) +{ + pktio_ops_i40e_data_t *pkt_i40e = pktio_entry->s.ops_data; + + return link_status_fd(pkt_i40e->sockfd, pkt_i40e->mdev.if_name); +} + +static void i40e_wait_link_up(pktio_entry_t *pktio_entry) +{ + while (!i40e_link_status(pktio_entry)) + sleep(1); +} + +static int i40e_capability(pktio_entry_t *pktio_entry, + odp_pktio_capability_t *capa) +{ + pktio_ops_i40e_data_t *pkt_i40e = pktio_entry->s.ops_data; + + *capa = pkt_i40e->capa; + return 0; +} + +static int i40e_input_queues_config(pktio_entry_t *pktio_entry, + const odp_pktin_queue_param_t *p) +{ + pktio_ops_i40e_data_t *pkt_i40e = pktio_entry->s.ops_data; + + if (p->op_mode == ODP_PKTIO_OP_MT_UNSAFE) + pkt_i40e->lockless_rx = 1; + else + pkt_i40e->lockless_rx = 0; + + return 0; +} + +static int i40e_output_queues_config(pktio_entry_t *pktio_entry, + const odp_pktout_queue_param_t *p) +{ + pktio_ops_i40e_data_t *pkt_i40e = pktio_entry->s.ops_data; + + if (p->op_mode == ODP_PKTIO_OP_MT_UNSAFE) + pkt_i40e->lockless_tx = 1; + else + pkt_i40e->lockless_tx = 0; + + return 0; +} + +static int i40e_mac_get(pktio_entry_t *pktio_entry, void *mac_addr) +{ + pktio_ops_i40e_data_t *pkt_i40e = pktio_entry->s.ops_data; + + if (mac_addr_get_fd(pkt_i40e->sockfd, pkt_i40e->mdev.if_name, + mac_addr) < 0) + return -1; + + return ETH_ALEN; +} + +static int i40e_init_global(void) +{ + ODP_PRINT("PKTIO: initialized " MODULE_NAME " interface\n"); + return 0; +} + +static pktio_ops_module_t i40e_pktio_ops = { + .base = { + .name = MODULE_NAME, + .init_global = i40e_init_global, + }, + + .open = i40e_open, + .close = i40e_close, + + .recv = i40e_recv, + .send = i40e_send, + + .link_status = i40e_link_status, + + .capability = i40e_capability, + + .mac_get = i40e_mac_get, + + .input_queues_config = i40e_input_queues_config, + .output_queues_config = i40e_output_queues_config, +}; + +/** i40e module entry point */ +ODP_MODULE_CONSTRUCTOR(netmap_pktio_ops) +{ + odp_module_constructor(&i40e_pktio_ops); + + odp_subsystem_register_module(pktio_ops, &i40e_pktio_ops); +} + +/* + * Temporary variable to enable link this module, + * will remove in Makefile scheme changes. + */ +int enable_link_i40e_pktio_ops; + +#endif /* ODP_MDEV */ diff --git a/platform/linux-generic/pktio/subsystem.c b/platform/linux-generic/pktio/subsystem.c index a3b36c144..d6d6c39d0 100644 --- a/platform/linux-generic/pktio/subsystem.c +++ b/platform/linux-generic/pktio/subsystem.c @@ -44,6 +44,9 @@ extern int enable_link_socket_mmap_pktio_ops; #ifdef ODP_PKTIO_TAP extern int enable_link_tap_pktio_ops; #endif +#ifdef ODP_MDEV +extern int enable_link_i40e_pktio_ops; +#endif ODP_SUBSYSTEM_CONSTRUCTOR(pktio_ops) { @@ -73,4 +76,7 @@ ODP_SUBSYSTEM_CONSTRUCTOR(pktio_ops) #ifdef ODP_PKTIO_TAP enable_link_tap_pktio_ops = 1; #endif +#ifdef ODP_MDEV + enable_link_i40e_pktio_ops = 1; +#endif } From patchwork Fri Jan 5 16:00:10 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Github ODP bot X-Patchwork-Id: 123535 Delivered-To: patch@linaro.org Received: by 10.140.22.227 with SMTP id 90csp985497qgn; Fri, 5 Jan 2018 08:04:18 -0800 (PST) X-Google-Smtp-Source: ACJfBovGKbFF0bVVGrHrH7KInjf7BoiaIqRc48wQl5gVJF11WYSoB+gIxZfci/717riQ6WrvtY9K X-Received: by 10.55.40.16 with SMTP id o16mr5023930qkh.216.1515168258439; Fri, 05 Jan 2018 08:04:18 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1515168258; cv=none; d=google.com; s=arc-20160816; b=oINg844I+/IyHDl5IVNkSSOrRZkxzCzvQoQjUko4ooTjqpFl6AlIGNvTvUHr676Y3e o7C8YkWEms6Wj4e1/vsmnKElCzmCUzC+w3Ybx34XVS7e9N9HpBFr3ziTwsJru080wpTk VE2gdyMBXBQfv1WnuYLfbJee6JqWFNiAllrGR8y7FPj32XSUK05tKQN3XaF8JAn9ukgw a4inQ4ACD3Qod4ZvCycTn9e/YasUQxeBCc9b4M/FsK3fubBPjRE+yPcX1WXJwkxenk7j rBS+da2+QlP+k3WWBiCgmKuiwbUFmu3RcDbq30y2cym0WRxvLuL20j0KdJrfezR4dk1J ScnA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:github-pr-num :references:in-reply-to:message-id:date:to:from:delivered-to :arc-authentication-results; bh=SB9pk3pf56C80wvhT2j81thnf/ZyukYhSUuD2Db+Jm0=; b=mKLdirrvK24T6gpxF492HzEMqiZr0v40r9Vb9TFYQsB+YTzjmLI/46tw3BjbWO8omk H0XZI2SPqzs/CBQmjIGlr37TPG8PCW7Dy73VDBIppK9oIZJJFzvXuO/QdoXwET7VmvSo m0smIYGafQUB4qy8dgnSg+4py6FdawHa8fYKItxVQcMqXerDS2q6nxMtOZ98AMThNqJG X7RKa/JqC4FYR5wlNF0xsYQRMfwAqNHyxt+PziLagMjlChpxZn+J0GQxIFkt7fRww+Rb +GTxMtg4yqaMvyYZSw5JIxoL9a9sxjgpbWJ/ginzaTFEP8Ay1OzzhuhpLJVT+qoI8j2B 2oGw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.197.127.237 as permitted sender) smtp.mailfrom=lng-odp-bounces@lists.linaro.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=yandex.ru Return-Path: Received: from lists.linaro.org (ec2-54-197-127-237.compute-1.amazonaws.com. [54.197.127.237]) by mx.google.com with ESMTP id c63si39154qkf.259.2018.01.05.08.04.18; Fri, 05 Jan 2018 08:04:18 -0800 (PST) Received-SPF: pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.197.127.237 as permitted sender) client-ip=54.197.127.237; Authentication-Results: mx.google.com; spf=pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.197.127.237 as permitted sender) smtp.mailfrom=lng-odp-bounces@lists.linaro.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=yandex.ru Received: by lists.linaro.org (Postfix, from userid 109) id 12E8761501; Fri, 5 Jan 2018 16:04:18 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on ip-10-142-244-252 X-Spam-Level: X-Spam-Status: No, score=-2.6 required=5.0 tests=BAYES_00,FREEMAIL_FROM, RCVD_IN_DNSWL_LOW, RCVD_IN_MSPIKE_H2, URIBL_BLOCKED autolearn=disabled version=3.4.0 Received: from [127.0.0.1] (localhost [127.0.0.1]) by lists.linaro.org (Postfix) with ESMTP id 5B0016150D; Fri, 5 Jan 2018 16:00:48 +0000 (UTC) X-Original-To: lng-odp@lists.linaro.org Delivered-To: lng-odp@lists.linaro.org Received: by lists.linaro.org (Postfix, from userid 109) id 04FEB60B11; Fri, 5 Jan 2018 16:00:22 +0000 (UTC) Received: from forward101o.mail.yandex.net (forward101o.mail.yandex.net [37.140.190.181]) by lists.linaro.org (Postfix) with ESMTPS id EF841609E7 for ; Fri, 5 Jan 2018 16:00:15 +0000 (UTC) Received: from mxback9j.mail.yandex.net (mxback9j.mail.yandex.net [IPv6:2a02:6b8:0:1619::112]) by forward101o.mail.yandex.net (Yandex) with ESMTP id 887D713426D7 for ; Fri, 5 Jan 2018 19:00:14 +0300 (MSK) Received: from smtp2o.mail.yandex.net (smtp2o.mail.yandex.net [2a02:6b8:0:1a2d::26]) by mxback9j.mail.yandex.net (nwsmtp/Yandex) with ESMTP id rkP4BcdG9r-0Ew4PZtn; Fri, 05 Jan 2018 19:00:14 +0300 Received: by smtp2o.mail.yandex.net (nwsmtp/Yandex) with ESMTPSA id wLSnxxib0I-0DY8JKeD; Fri, 05 Jan 2018 19:00:13 +0300 (using TLSv1.2 with cipher ECDHE-RSA-AES128-SHA256 (128/128 bits)) (Client certificate not present) From: Github ODP bot To: lng-odp@lists.linaro.org Date: Fri, 5 Jan 2018 19:00:10 +0300 Message-Id: <1515168010-29173-4-git-send-email-odpbot@yandex.ru> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1515168010-29173-1-git-send-email-odpbot@yandex.ru> References: <1515168010-29173-1-git-send-email-odpbot@yandex.ru> Github-pr-num: 383 Subject: [lng-odp] [PATCH CATERPILLAR v1 3/3] linux-gen: add cxgb4 mediated device driver X-BeenThere: lng-odp@lists.linaro.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: "The OpenDataPlane \(ODP\) List" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: lng-odp-bounces@lists.linaro.org Sender: "lng-odp" From: Ilias Apalodimas Signed-off-by: Ilias Apalodimas Signed-off-by: Mykyta Iziumtsev --- /** Email created from pull request 383 (MykytaI:caterpillar_mdev_cxgb4) ** https://github.com/Linaro/odp/pull/383 ** Patch: https://github.com/Linaro/odp/pull/383.patch ** Base sha: 4d17f8ae64aba0e6f24877be30f86ae5880cef7e ** Merge commit sha: 0e6e747c9f2997442be64a3cc70d7f117f29216d **/ platform/linux-generic/Makefile.am | 1 + platform/linux-generic/pktio/mdev/cxgb4.c | 894 ++++++++++++++++++++++++++++++ platform/linux-generic/pktio/subsystem.c | 2 + 3 files changed, 897 insertions(+) create mode 100644 platform/linux-generic/pktio/mdev/cxgb4.c diff --git a/platform/linux-generic/Makefile.am b/platform/linux-generic/Makefile.am index 85f902115..d3cff2f9f 100644 --- a/platform/linux-generic/Makefile.am +++ b/platform/linux-generic/Makefile.am @@ -259,6 +259,7 @@ __LIB__libodp_linux_la_SOURCES = \ pktio/ring.c \ pktio/mdev.c \ pktio/mdev/i40e.c \ + pktio/mdev/cxgb4.c \ pool/generic.c \ pool/subsystem.c \ odp_pkt_queue.c \ diff --git a/platform/linux-generic/pktio/mdev/cxgb4.c b/platform/linux-generic/pktio/mdev/cxgb4.c new file mode 100644 index 000000000..a6260722b --- /dev/null +++ b/platform/linux-generic/pktio/mdev/cxgb4.c @@ -0,0 +1,894 @@ +/*Copyright (c) 2018, Linaro Limited + * All rights reserved. + * + * SPDX-License-Identifier: BSD-3-Clause + */ + +#include "config.h" + +#ifdef ODP_MDEV + +#include + +#include +#include +#include +#include +#include +#include + +#include +#include +#include + +#include +#include + +#include + +#include + +#include +#include +#include +#include +#include + +#define MODULE_NAME "cxgb4" + +#define CXGB4_TX_BUF_SIZE 2048U + +#define CXGB4_TX_INLINE_MAX (256 - sizeof(cxgb4_fw_eth_tx_pkt_wr_t) \ + - sizeof(cxgb4_cpl_tx_pkt_core_t)) + +/* RX queue definitions */ +#define CXGB4_RX_QUEUE_NUM_MAX 32 + +/** RX descriptor */ +typedef struct { + uint32_t padding[12]; + + odp_u32be_t hdrbuflen_pidx; +#define RX_DESC_NEW_BUF_FLAG (1U << 31) + odp_u32be_t pldbuflen_qid; + union { +#define RX_DESC_GEN_SHIFT 7 +#define RX_DESC_GEN_MASK 0x1 +#define RX_DESC_TYPE_SHIFT 4 +#define RX_DESC_TYPE_MASK 0x3 +#define RX_DESC_TYPE_FLBUF_X 0 +#define RX_DESC_TYPE_CPL_X 1 +#define RX_DESC_TYPE_INTR_X 2 + uint8_t type_gen; +#define RX_DESC_TIMESTAMP_MASK 0xfffffffffffffffULL + odp_u64be_t last_flit; + }; +} cxgb4_rx_desc_t; + +#define RX_DESC_TO_GEN(rxd) \ + (((rxd)->type_gen >> RX_DESC_GEN_SHIFT) & RX_DESC_GEN_MASK) +#define RX_DESC_TO_TYPE(rxd) \ + (((rxd)->type_gen >> RX_DESC_TYPE_SHIFT) & RX_DESC_TYPE_MASK) + +/** RX queue data */ +typedef struct { + cxgb4_rx_desc_t *rx_descs; /**< RX queue base */ + + odp_u32le_t *doorbell_fl; /**< Free list refill doorbell */ + odp_u32le_t *doorbell_desc; /**< Rx descriptor free doorbell */ + uint32_t doorbell_fl_key; /**< 'Key' to the doorbell */ + uint32_t doorbell_desc_key; /**< 'Key' to the doorbell */ + + uint16_t rx_queue_len; /**< Number of RX desc entries */ + uint16_t rx_next; /**< Next RX desc to handle */ + + uint32_t gen:1; /**< RX queue generation */ + + odp_u64be_t *free_list; /**< Free list base */ + + uint8_t free_list_len; /**< Number of free list entries */ + uint8_t commit_pending; /**< Free list entries pending commit */ + + uint8_t cidx; /**< Free list consumer index */ + uint8_t pidx; /**< Free list producer index */ + + uint32_t offset; /**< Offset into last free fragment */ + + mdev_dma_area_t rx_data; /**< RX packet payload area */ + + odp_ticketlock_t lock; /**< RX queue lock */ +} cxgb4_rx_queue_t ODP_ALIGNED_CACHE; + +/* TX queue definitions */ +#define CXGB4_TX_QUEUE_NUM_MAX 32 + +typedef struct { + odp_u64be_t data[8]; +} cxgb4_tx_desc_t; + +typedef struct { +#define CXGB4_FW_ETH_TX_PKT_WR 0x08000000UL + odp_u32be_t op_immdlen; + odp_u32be_t equiq_to_len16; + odp_u64be_t r3; +} cxgb4_fw_eth_tx_pkt_wr_t; + +typedef struct { +#define CPL_TX_PKT_XT 0xEE000000UL +#define TXPKT_PF_S 8 +#define TXPKT_PF_V(x) ((x) << TXPKT_PF_S) +#define TXPKT_INTF_S 16 +#define TXPKT_INTF_V(x) ((x) << TXPKT_INTF_S) + odp_u32be_t ctrl0; + odp_u16be_t pack; + odp_u16be_t len; +#define TXPKT_IPCSUM_DIS_F (1UL << 62) +#define TXPKT_L4CSUM_DIS_F (1UL << 63) + odp_u64be_t ctrl1; +} cxgb4_cpl_tx_pkt_core_t; + +typedef struct { + odp_u32be_t len[2]; + odp_u64be_t addr[2]; +} cxgb4_sg_pair_t; + +typedef struct { +#define CXGB4_ULP_TX_SC_DSGL (0x82UL << 24) + odp_u32be_t sg_pairs_num; + odp_u32be_t len0; + odp_u64be_t addr0; + cxgb4_sg_pair_t sg_pairs[0]; +} cxgb4_sg_list_t; + +typedef struct { + odp_u32be_t qid; + odp_u16be_t cidx; + odp_u16be_t pidx; +} cxgb4_tx_queue_stats; + +/** TX queue data */ +typedef struct { + cxgb4_tx_desc_t *tx_descs; /**< TX queue base */ + cxgb4_tx_queue_stats *stats; /**< TX queue stats */ + + odp_u32le_t *doorbell_desc; /**< TX queue doorbell */ + uint32_t doorbell_desc_key; /**< 'Key' to the doorbell */ + + uint16_t tx_queue_len; /**< Number of TX desc entries */ + uint16_t tx_next; /**< Next TX desc to insert */ + + mdev_dma_area_t tx_data; /**< TX packet payload area */ + + odp_ticketlock_t lock; /**< TX queue lock */ +} cxgb4_tx_queue_t ODP_ALIGNED_CACHE; + +/** Packet socket using mediated cxgb4 device */ +typedef struct { + /** RX queue hot data */ + cxgb4_rx_queue_t rx_queues[CXGB4_RX_QUEUE_NUM_MAX]; + + /** TX queue hot data */ + cxgb4_tx_queue_t tx_queues[CXGB4_TX_QUEUE_NUM_MAX]; + + odp_pool_t pool; /**< pool to alloc packets from */ + + odp_bool_t lockless_rx; /**< no locking for RX */ + uint16_t free_list_align; /**< Alignment required for RX chunks */ + + odp_bool_t lockless_tx; /**< no locking for TX */ + uint8_t tx_channel; /**< TX channel of the interface */ + uint8_t phys_function; /**< Physical function of the interface */ + + odp_pktio_capability_t capa; /**< interface capabilities */ + + uint8_t *mmio; /**< MMIO region */ + + int sockfd; /**< control socket */ + + mdev_device_t mdev; /**< Common mdev data */ +} pktio_ops_cxgb4_data_t; + +static void cxgb4_rx_refill(cxgb4_rx_queue_t *rxq, uint8_t num); +static void cxgb4_wait_link_up(pktio_entry_t *pktio_entry); +static int cxgb4_close(pktio_entry_t *pktio_entry); + +static int cxgb4_mmio_register(pktio_ops_cxgb4_data_t *pkt_cxgb4, + uint64_t offset, uint64_t size) +{ + ODP_ASSERT(pkt_cxgb4->mmio == NULL); + + pkt_cxgb4->mmio = mdev_region_mmap(&pkt_cxgb4->mdev, offset, size); + if (pkt_cxgb4->mmio == MAP_FAILED) { + ODP_ERR("Cannot mmap MMIO\n"); + return -1; + } + + ODP_DBG("Register MMIO region: 0x%llx@%016llx\n", size, offset); + + return 0; +} + +static int cxgb4_rx_queue_register(pktio_ops_cxgb4_data_t *pkt_cxgb4, + uint64_t offset, uint64_t size, + uint64_t free_list_offset) +{ + uint16_t rxq_idx = pkt_cxgb4->capa.max_input_queues++; + cxgb4_rx_queue_t *rxq = &pkt_cxgb4->rx_queues[rxq_idx]; + struct ethtool_ringparam ering; + uint64_t val; + int ret; + + ODP_ASSERT(rxq_idx < ARRAY_SIZE(pkt_cxgb4->rx_queues)); + + odp_ticketlock_init(&rxq->lock); + + ret = ethtool_ringparam_get_fd(pkt_cxgb4->sockfd, + pkt_cxgb4->mdev.if_name, &ering); + if (ret) { + ODP_ERR("Cannot get queue length\n"); + return -1; + } + rxq->rx_queue_len = ering.rx_mini_pending; + rxq->free_list_len = ering.rx_pending + 8; + + ret = sysfs_attr_u64_get(&val, "/sys/class/net/%s/queues/rx-%u/cxgb4/" + "doorbell_fl_offset", + pkt_cxgb4->mdev.if_name, rxq_idx); + if (ret) { + ODP_ERR("Cannot get %s rx-%u doorbell_fl_offset\n", + pkt_cxgb4->mdev.if_name, rxq_idx); + return -1; + } + rxq->doorbell_fl = (odp_u32le_t *)(pkt_cxgb4->mmio + val); + + ret = sysfs_attr_u64_get(&val, "/sys/class/net/%s/queues/rx-%u/cxgb4/" + "doorbell_fl_key", + pkt_cxgb4->mdev.if_name, rxq_idx); + if (ret) { + ODP_ERR("Cannot get %s rx-%u doorbell_fl_key\n", + pkt_cxgb4->mdev.if_name, rxq_idx); + return -1; + } + rxq->doorbell_fl_key = val; + + ret = sysfs_attr_u64_get(&val, "/sys/class/net/%s/queues/rx-%u/cxgb4/" + "doorbell_desc_offset", + pkt_cxgb4->mdev.if_name, rxq_idx); + if (ret) { + ODP_ERR("Cannot get %s rx-%u doorbell_desc_offset\n", + pkt_cxgb4->mdev.if_name, rxq_idx); + return -1; + } + rxq->doorbell_desc = (odp_u32le_t *)(pkt_cxgb4->mmio + val); + + ret = sysfs_attr_u64_get(&val, "/sys/class/net/%s/queues/rx-%u/cxgb4/" + "doorbell_desc_key", + pkt_cxgb4->mdev.if_name, rxq_idx); + if (ret) { + ODP_ERR("Cannot get %s rx-%u doorbell_desc_key\n", + pkt_cxgb4->mdev.if_name, rxq_idx); + return -1; + } + rxq->doorbell_desc_key = val; + + ODP_ASSERT(rxq->rx_queue_len * sizeof(*rxq->rx_descs) <= size); + + rxq->rx_descs = mdev_region_mmap(&pkt_cxgb4->mdev, offset, size); + if (rxq->rx_descs == MAP_FAILED) { + ODP_ERR("Cannot mmap RX queue\n"); + return -1; + } + + ODP_ASSERT(rxq->free_list_len * sizeof(*rxq->free_list) <= + ODP_PAGE_SIZE); + + rxq->free_list = + mdev_region_mmap(&pkt_cxgb4->mdev, free_list_offset, ODP_PAGE_SIZE); + if (rxq->free_list == MAP_FAILED) { + ODP_ERR("Cannot mmap RX queue free list\n"); + return -1; + } + + rxq->rx_data.size = rxq->free_list_len * ODP_PAGE_SIZE; + ret = mdev_dma_area_alloc(&pkt_cxgb4->mdev, &rxq->rx_data); + if (ret) { + ODP_ERR("Cannot allocate RX queue DMA area\n"); + return -1; + } + + rxq->gen = 1; + + /* + * Leave 1 HW block (8 entries) unpopulated, + * otherwise HW will think the free list is empty. + */ + cxgb4_rx_refill(rxq, rxq->free_list_len - 8); + rxq->cidx = rxq->free_list_len - 1; + + ODP_DBG("Register RX queue region: 0x%llx@%016llx\n", size, offset); + ODP_DBG(" RX descriptors: %u\n", rxq->rx_queue_len); + ODP_DBG(" RX free list entries: %u\n", rxq->free_list_len); + + return 0; +} + +static int cxgb4_tx_queue_register(pktio_ops_cxgb4_data_t *pkt_cxgb4, + uint64_t offset, uint64_t size) +{ + uint16_t txq_idx = pkt_cxgb4->capa.max_output_queues++; + cxgb4_tx_queue_t *txq = &pkt_cxgb4->tx_queues[txq_idx]; + struct ethtool_ringparam ering; + uint64_t val; + int ret; + + ODP_ASSERT(txq_idx < ARRAY_SIZE(pkt_cxgb4->tx_queues)); + + odp_ticketlock_init(&txq->lock); + + ret = ethtool_ringparam_get_fd(pkt_cxgb4->sockfd, + pkt_cxgb4->mdev.if_name, &ering); + if (ret) { + ODP_ERR("Cannot get queue length\n"); + return -1; + } + txq->tx_queue_len = ering.tx_pending; + + ret = sysfs_attr_u64_get(&val, "/sys/class/net/%s/queues/tx-%u/cxgb4/" + "doorbell_desc_offset", + pkt_cxgb4->mdev.if_name, txq_idx); + if (ret) { + ODP_ERR("Cannot get %s tx-%u doorbell_desc_offset\n", + pkt_cxgb4->mdev.if_name, txq_idx); + return -1; + } + txq->doorbell_desc = (odp_u32le_t *)(pkt_cxgb4->mmio + val); + + ret = sysfs_attr_u64_get(&val, "/sys/class/net/%s/queues/tx-%u/cxgb4/" + "doorbell_desc_key", + pkt_cxgb4->mdev.if_name, txq_idx); + if (ret) { + ODP_ERR("Cannot get %s tx-%u doorbell_desc_key\n", + pkt_cxgb4->mdev.if_name, txq_idx); + return -1; + } + txq->doorbell_desc_key = val; + + ODP_ASSERT(txq->tx_queue_len * sizeof(*txq->tx_descs) + + sizeof(*txq->stats) <= size); + + txq->tx_descs = mdev_region_mmap(&pkt_cxgb4->mdev, offset, size); + if (txq->tx_descs == MAP_FAILED) { + ODP_ERR("Cannot mmap TX queue\n"); + return -1; + } + + txq->stats = + (cxgb4_tx_queue_stats *)(txq->tx_descs + txq->tx_queue_len); + + txq->tx_data.size = txq->tx_queue_len * CXGB4_TX_BUF_SIZE; + ret = mdev_dma_area_alloc(&pkt_cxgb4->mdev, &txq->tx_data); + if (ret) { + ODP_ERR("Cannot allocate TX queue DMA area\n"); + return -1; + } + + ODP_DBG("Register TX queue region: 0x%llx@%016llx\n", size, offset); + ODP_DBG(" TX descriptors: %u\n", txq->tx_queue_len); + + return 0; +} + +static int cxgb4_region_info_cb(mdev_device_t *mdev, + struct vfio_region_info *region_info) +{ + pktio_ops_cxgb4_data_t *pkt_cxgb4 = + odp_container_of(mdev, pktio_ops_cxgb4_data_t, mdev); + mdev_region_class_t class_info; + + if (vfio_get_region_cap_type(region_info, &class_info) < 0) { + ODP_ERR("Cannot find class_info in region %u\n", + region_info->index); + return -1; + } + + switch (class_info.type) { + case VFIO_NET_MMIO: + return cxgb4_mmio_register(pkt_cxgb4, + region_info->offset, + region_info->size); + + case VFIO_NET_DESCRIPTORS: + if (class_info.subtype == VFIO_NET_MDEV_RX) { + struct vfio_region_info_cap_sparse_mmap *sparse; + + if (vfio_get_region_sparse_mmaps(region_info, + &sparse) < 0) { + ODP_ERR("RX queue in region %u: %s\n", + region_info->index, + "no areas found"); + return -1; + } + + if (sparse->nr_areas != 2) { + ODP_ERR("RX queue in region %u: %s\n", + region_info->index, + "wrong number of areas"); + return -1; + } + + ODP_ASSERT(sparse->areas[1].size == ODP_PAGE_SIZE); + + return cxgb4_rx_queue_register(pkt_cxgb4, + sparse->areas[0].offset, + sparse->areas[0].size, + sparse->areas[1].offset); + } + if (class_info.subtype == VFIO_NET_MDEV_TX) + return cxgb4_tx_queue_register(pkt_cxgb4, + region_info->offset, + region_info->size); + /* fallthrough */ + + default: + ODP_ERR("Unexpected region %u (class %u:%u)\n", + region_info->index, class_info.type, + class_info.subtype); + return -1; + } +} + +static int cxgb4_open(odp_pktio_t id ODP_UNUSED, + pktio_entry_t *pktio_entry, + const char *resource, odp_pool_t pool) +{ + pktio_ops_cxgb4_data_t *pkt_cxgb4; + uint64_t val; + int ret; + + ODP_ASSERT(pool != ODP_POOL_INVALID); + + if (strncmp(resource, NET_MDEV_PREFIX, strlen(NET_MDEV_PREFIX))) + return -1; + + ODP_DBG("%s: probing resource %s\n", MODULE_NAME, resource); + + pkt_cxgb4 = ODP_OPS_DATA_ALLOC(sizeof(*pkt_cxgb4)); + if (odp_unlikely(pkt_cxgb4 == NULL)) { + ODP_ERR("Failed to allocate pktio_ops_cxgb4_data_t struct"); + return -1; + } + pktio_entry->s.ops_data = pkt_cxgb4; + + memset(pkt_cxgb4, 0, sizeof(*pkt_cxgb4)); + + pkt_cxgb4->sockfd = socket(AF_INET, SOCK_DGRAM, 0); + if (pkt_cxgb4->sockfd == -1) { + ODP_ERR("Cannot get device control socket\n"); + goto out; + } + + ret = + mdev_device_create(&pkt_cxgb4->mdev, MODULE_NAME, + resource + strlen(NET_MDEV_PREFIX), + cxgb4_region_info_cb); + if (ret) + goto out; + + pkt_cxgb4->pool = pool; + + ret = sysfs_attr_u64_get(&val, "/sys/class/net/%s/tx_channel", + pkt_cxgb4->mdev.if_name); + if (ret) { + ODP_ERR("Cannot get %s tx_channel\n", pkt_cxgb4->mdev.if_name); + return -1; + } + pkt_cxgb4->tx_channel = val; + + ret = sysfs_attr_u64_get(&val, "/sys/class/net/%s/phys_function", + pkt_cxgb4->mdev.if_name); + if (ret) { + ODP_ERR("Cannot get %s phys_function\n", + pkt_cxgb4->mdev.if_name); + return -1; + } + pkt_cxgb4->phys_function = val; + + ret = sysfs_attr_u64_get(&val, "/sys/class/net/%s/free_list_align", + pkt_cxgb4->mdev.if_name); + if (ret) { + ODP_ERR("Cannot get %s free_list_align\n", + pkt_cxgb4->mdev.if_name); + return -1; + } + pkt_cxgb4->free_list_align = val; + + cxgb4_wait_link_up(pktio_entry); + + ODP_DBG("%s: open %s is successful\n", MODULE_NAME, + pkt_cxgb4->mdev.if_name); + + return 0; + +out: + cxgb4_close(pktio_entry); + return -1; +} + +static int cxgb4_close(pktio_entry_t *pktio_entry) +{ + pktio_ops_cxgb4_data_t *pkt_cxgb4 = pktio_entry->s.ops_data; + + ODP_DBG("%s: close %s\n", MODULE_NAME, pkt_cxgb4->mdev.if_name); + + mdev_device_destroy(&pkt_cxgb4->mdev); + + for (uint16_t i = 0; i < pkt_cxgb4->capa.max_input_queues; i++) { + cxgb4_rx_queue_t *rxq = &pkt_cxgb4->rx_queues[i]; + + if (rxq->rx_data.size) + mdev_dma_area_free(&pkt_cxgb4->mdev, &rxq->rx_data); + } + + for (uint16_t i = 0; i < pkt_cxgb4->capa.max_output_queues; i++) { + cxgb4_tx_queue_t *txq = &pkt_cxgb4->tx_queues[i]; + + if (txq->tx_data.size) + mdev_dma_area_free(&pkt_cxgb4->mdev, &txq->tx_data); + } + + if (pkt_cxgb4->sockfd != -1) + close(pkt_cxgb4->sockfd); + + ODP_OPS_DATA_FREE(pkt_cxgb4); + + return 0; +} + +static void cxgb4_rx_refill(cxgb4_rx_queue_t *rxq, uint8_t num) +{ + rxq->commit_pending += num; + + while (num) { + uint64_t iova = rxq->rx_data.iova + rxq->pidx * ODP_PAGE_SIZE; + + rxq->free_list[rxq->pidx] = odp_cpu_to_be_64(iova); + + rxq->pidx++; + if (odp_unlikely(rxq->pidx >= rxq->free_list_len)) + rxq->pidx = 0; + + num--; + } + + /* We commit free list entries to HW in packs of 8 */ + if (rxq->commit_pending >= 8) { + uint32_t val = rxq->doorbell_fl_key | (rxq->commit_pending / 8); + + /* Ring the doorbell */ + odpdrv_mmio_u32le_write(val, rxq->doorbell_fl); + + rxq->commit_pending &= 7; + } +} + +static int cxgb4_recv(pktio_entry_t *pktio_entry, + int rxq_idx, odp_packet_t pkt_table[], int num) +{ + pktio_ops_cxgb4_data_t *pkt_cxgb4 = pktio_entry->s.ops_data; + cxgb4_rx_queue_t *rxq = &pkt_cxgb4->rx_queues[rxq_idx]; + uint16_t refill_count = 0; + int rx_pkts = 0; + + if (!pkt_cxgb4->lockless_rx) + odp_ticketlock_lock(&rxq->lock); + + while (rx_pkts < num) { + volatile cxgb4_rx_desc_t *rxd = &rxq->rx_descs[rxq->rx_next]; + odp_packet_t pkt; + uint32_t pkt_len; + uint8_t type; + + if (RX_DESC_TO_GEN(rxd) != rxq->gen) + break; + + type = RX_DESC_TO_TYPE(rxd); + + if (odp_unlikely(type != RX_DESC_TYPE_FLBUF_X)) { + ODP_ERR("Invalid rxd type %u\n", type); + + rxq->rx_next++; + if (odp_unlikely(rxq->rx_next >= rxq->rx_queue_len)) { + rxq->rx_next = 0; + rxq->gen ^= 1; + } + + continue; + } + + pkt_len = odp_be_to_cpu_32(rxd->pldbuflen_qid); + + /* + * HW skips trailing area in current RX buffer and starts in the + * next one from the beginning. + */ + if (pkt_len & RX_DESC_NEW_BUF_FLAG) { + rxq->cidx++; + if (odp_unlikely(rxq->cidx >= rxq->free_list_len)) + rxq->cidx = 0; + + rxq->offset = 0; + refill_count++; + + pkt_len ^= RX_DESC_NEW_BUF_FLAG; + } + + if (odp_unlikely(rxq->offset + pkt_len > ODP_PAGE_SIZE)) { + /* TODO: reset the HW and reinit ? */ + ODP_ABORT("Packet write beyond buffer boundary\n"); + } + + pkt = odp_packet_alloc(pkt_cxgb4->pool, pkt_len); + if (odp_likely(pkt != ODP_PACKET_INVALID)) { + odp_packet_hdr_t *pkt_hdr; + + /* + * NIC is aligning L2 header on 4-byte boundary, hence + * we need to strip 2 leading bytes. + */ + odp_packet_copy_from_mem(pkt, 0, pkt_len - 2, + (uint8_t *)rxq->rx_data.vaddr + + rxq->cidx * ODP_PAGE_SIZE + + rxq->offset + 2); + + pkt_hdr = odp_packet_hdr(pkt); + pkt_hdr->input = pktio_entry->s.handle; + + pkt_table[rx_pkts] = pkt; + rx_pkts++; + } + + rxq->offset += + ROUNDUP_ALIGN(pkt_len, pkt_cxgb4->free_list_align); + + rxq->rx_next++; + if (odp_unlikely(rxq->rx_next >= rxq->rx_queue_len)) { + rxq->rx_next = 0; + rxq->gen ^= 1; + } + } + + if (rx_pkts) + odpdrv_mmio_u32le_write(rx_pkts | rxq->doorbell_desc_key, + rxq->doorbell_desc); + + if (refill_count) + cxgb4_rx_refill(rxq, refill_count); + + if (!pkt_cxgb4->lockless_rx) + odp_ticketlock_unlock(&rxq->lock); + + return rx_pkts; +} + +static int cxgb4_send(pktio_entry_t *pktio_entry, + int txq_idx, const odp_packet_t pkt_table[], int num) +{ + pktio_ops_cxgb4_data_t *pkt_cxgb4 = pktio_entry->s.ops_data; + cxgb4_tx_queue_t *txq = &pkt_cxgb4->tx_queues[txq_idx]; + uint16_t budget, tx_txds = 0; + int tx_pkts = 0; + + if (!pkt_cxgb4->lockless_tx) + odp_ticketlock_lock(&txq->lock); + + /* Determine how many packets will fit in TX queue */ + budget = txq->tx_queue_len - 1; + budget -= txq->tx_next; + budget += odp_be_to_cpu_16(txq->stats->cidx); + budget &= txq->tx_queue_len - 1; + + while (tx_txds < budget && tx_pkts < num) { + uint16_t pkt_len = _odp_packet_len(pkt_table[tx_pkts]); + + cxgb4_tx_desc_t *txd = &txq->tx_descs[txq->tx_next]; + uint32_t txd_len; + + cxgb4_fw_eth_tx_pkt_wr_t *wr; + cxgb4_cpl_tx_pkt_core_t *cpl; + uint32_t op_immdlen; + + /* Skip oversized packets silently */ + if (odp_unlikely(pkt_len > CXGB4_TX_BUF_SIZE)) { + tx_pkts++; + continue; + } + + wr = (cxgb4_fw_eth_tx_pkt_wr_t *)txd; + cpl = (cxgb4_cpl_tx_pkt_core_t *)(wr + 1); + + txd_len = sizeof(*wr) + sizeof(*cpl); + op_immdlen = CXGB4_FW_ETH_TX_PKT_WR | sizeof(*cpl); + + if (pkt_len <= CXGB4_TX_INLINE_MAX) { + uint8_t *pos; + uint32_t left; + + txd_len += pkt_len; + + /* Try next time */ + if (tx_txds + DIV_ROUND_UP(txd_len, sizeof(*txd)) > + budget) + break; + + op_immdlen += pkt_len; + + /* Packet copying shall take care of wrapping */ + pos = (uint8_t *)(cpl + 1); + left = (uint8_t *)(txq->tx_descs + txq->tx_queue_len) - + pos; + + if (odp_likely(left >= pkt_len)) { + odp_packet_copy_to_mem(pkt_table[tx_pkts], 0, + pkt_len, pos); + } else { + odp_packet_copy_to_mem(pkt_table[tx_pkts], 0, + left, pos); + odp_packet_copy_to_mem(pkt_table[tx_pkts], left, + pkt_len - left, + &txq->tx_descs[0]); + } + } else { + uint32_t offset = txq->tx_next * CXGB4_TX_BUF_SIZE; + cxgb4_sg_list_t *sgl; + + txd_len += sizeof(*sgl); + + odp_packet_copy_to_mem(pkt_table[tx_pkts], 0, pkt_len, + (uint8_t *)txq->tx_data.vaddr + + offset); + + sgl = (cxgb4_sg_list_t *)(cpl + 1); + + sgl->sg_pairs_num = + odp_cpu_to_be_32(CXGB4_ULP_TX_SC_DSGL | 1); + sgl->addr0 = + odp_cpu_to_be_64(txq->tx_data.iova + offset); + sgl->len0 = odp_cpu_to_be_32(pkt_len); + } + + wr->op_immdlen = odp_cpu_to_be_32(op_immdlen); + wr->equiq_to_len16 = + odp_cpu_to_be_32(DIV_ROUND_UP(txd_len, 16)); + wr->r3 = odp_cpu_to_be_64(0); + + cpl->ctrl0 = + odp_cpu_to_be_32(CPL_TX_PKT_XT | + TXPKT_INTF_V(pkt_cxgb4->tx_channel) | + TXPKT_PF_V(pkt_cxgb4->phys_function)); + cpl->pack = odp_cpu_to_be_16(0); + cpl->len = odp_cpu_to_be_16(pkt_len); + cpl->ctrl1 = + odp_cpu_to_be_64(TXPKT_L4CSUM_DIS_F | TXPKT_IPCSUM_DIS_F); + + txq->tx_next += DIV_ROUND_UP(txd_len, sizeof(*txd)); + if (odp_unlikely(txq->tx_next >= txq->tx_queue_len)) + txq->tx_next -= txq->tx_queue_len; + + tx_txds += DIV_ROUND_UP(txd_len, sizeof(*txd)); + tx_pkts++; + } + + /* Ring the doorbell */ + odpdrv_mmio_u32le_write(txq->doorbell_desc_key | tx_txds, + txq->doorbell_desc); + + if (!pkt_cxgb4->lockless_tx) + odp_ticketlock_unlock(&txq->lock); + + odp_packet_free_multi(pkt_table, tx_pkts); + + return tx_pkts; +} + +static int cxgb4_link_status(pktio_entry_t *pktio_entry) +{ + pktio_ops_cxgb4_data_t *pkt_cxgb4 = pktio_entry->s.ops_data; + + return link_status_fd(pkt_cxgb4->sockfd, pkt_cxgb4->mdev.if_name); +} + +static void cxgb4_wait_link_up(pktio_entry_t *pktio_entry) +{ + while (!cxgb4_link_status(pktio_entry)) + sleep(1); +} + +static int cxgb4_capability(pktio_entry_t *pktio_entry, + odp_pktio_capability_t *capa) +{ + pktio_ops_cxgb4_data_t *pkt_cxgb4 = pktio_entry->s.ops_data; + + *capa = pkt_cxgb4->capa; + return 0; +} + +static int cxgb4_input_queues_config(pktio_entry_t *pktio_entry, + const odp_pktin_queue_param_t *p) +{ + pktio_ops_cxgb4_data_t *pkt_cxgb4 = pktio_entry->s.ops_data; + + if (p->op_mode == ODP_PKTIO_OP_MT_UNSAFE) + pkt_cxgb4->lockless_rx = 1; + else + pkt_cxgb4->lockless_rx = 0; + + return 0; +} + +static int cxgb4_output_queues_config(pktio_entry_t *pktio_entry, + const odp_pktout_queue_param_t *p) +{ + pktio_ops_cxgb4_data_t *pkt_cxgb4 = pktio_entry->s.ops_data; + + if (p->op_mode == ODP_PKTIO_OP_MT_UNSAFE) + pkt_cxgb4->lockless_tx = 1; + else + pkt_cxgb4->lockless_tx = 0; + + return 0; +} + +static int cxgb4_mac_get(pktio_entry_t *pktio_entry, void *mac_addr) +{ + pktio_ops_cxgb4_data_t *pkt_cxgb4 = pktio_entry->s.ops_data; + + if (mac_addr_get_fd(pkt_cxgb4->sockfd, pkt_cxgb4->mdev.if_name, + mac_addr) < 0) + return -1; + + return ETH_ALEN; +} + +static int cxgb4_init_global(void) +{ + ODP_PRINT("PKTIO: initialized " MODULE_NAME " interface\n"); + return 0; +} + +static pktio_ops_module_t cxgb4_pktio_ops = { + .base = { + .name = MODULE_NAME, + .init_global = cxgb4_init_global, + }, + + .open = cxgb4_open, + .close = cxgb4_close, + + .recv = cxgb4_recv, + .send = cxgb4_send, + + .link_status = cxgb4_link_status, + + .capability = cxgb4_capability, + + .mac_get = cxgb4_mac_get, + + .input_queues_config = cxgb4_input_queues_config, + .output_queues_config = cxgb4_output_queues_config, +}; + +/** cxgb4 module entry point */ +ODP_MODULE_CONSTRUCTOR(cxgb4_module_init) +{ + odp_module_constructor(&cxgb4_pktio_ops); + + odp_subsystem_register_module(pktio_ops, &cxgb4_pktio_ops); +} + +/* + * Temporary variable to enable link this module, + * will remove in Makefile scheme changes. + */ +int enable_link_cxgb4_pktio_ops; + +#endif /* ODP_MDEV */ diff --git a/platform/linux-generic/pktio/subsystem.c b/platform/linux-generic/pktio/subsystem.c index d6d6c39d0..8c18ef97f 100644 --- a/platform/linux-generic/pktio/subsystem.c +++ b/platform/linux-generic/pktio/subsystem.c @@ -46,6 +46,7 @@ extern int enable_link_tap_pktio_ops; #endif #ifdef ODP_MDEV extern int enable_link_i40e_pktio_ops; +extern int enable_link_cxgb4_pktio_ops; #endif ODP_SUBSYSTEM_CONSTRUCTOR(pktio_ops) @@ -78,5 +79,6 @@ ODP_SUBSYSTEM_CONSTRUCTOR(pktio_ops) #endif #ifdef ODP_MDEV enable_link_i40e_pktio_ops = 1; + enable_link_cxgb4_pktio_ops = 1; #endif }