From patchwork Thu Jul 22 11:02:50 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Boris Pismenny X-Patchwork-Id: 485112 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-20.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER, INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B8512C63793 for ; Thu, 22 Jul 2021 11:04:37 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 942916120D for ; Thu, 22 Jul 2021 11:04:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231588AbhGVKYA (ORCPT ); Thu, 22 Jul 2021 06:24:00 -0400 Received: from mail-mw2nam12on2074.outbound.protection.outlook.com ([40.107.244.74]:34017 "EHLO NAM12-MW2-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S231585AbhGVKXq (ORCPT ); Thu, 22 Jul 2021 06:23:46 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=HHbeA19Wjzifw05B7ElJ6WYIiHhtVBa/6YaCTUQb1fMUn90gFirb64mnrO7GPsaaho7wp7z+SPLZE1wkZwZ7uX6+LIRpeFFrUYQSIWd6m3Q2kFfFx4oR75D6Mv1LsPcR7nxtN5tmsyB3mKYZdsQ9L6IUCFI0dh3xNW2INEmImw2/R+0I0U3LFgFVLV/U7n15Pw9+UoEwSCyTQaTK2SWu38PB6+5Q7vXFYIjVQlS2obfqFW+4KxOgr8j6hkfZewA61kc7x6ur7OliVaNaLzGmhOAFjeBLHQZxmS1Hjhw5WJsr570qyOejXN1MNWFGPvrQKFcxqp3rZgbwg2iuJWxQww== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=QqTAAYwXQwigIi/k4FYFVIXpI4MtI+tQdOwA2kOIUx0=; b=BXwx902eofvQn4vkxVPGysoeO+19fcE1d3NIQ2gOqawGYxiyXlO7v+woijJeZQpA3RoUyrY9dgozEHN3oaogmss7tK3VMS10UsOcl0UXiGKjN9qzoiUd+55rgV6HN76KprRiiPLf9HqYrCk0xxuh+sdbAy4zdbjwyLrIUbNBsO15loj3RaaDv6W9VAuysYXBd5Ja5jLJmCrNtKR8LQHF1szhhf2D/UUjugK3x4gg5H0V1fHTttfwfLS6y/wIkrkPEqqrsVu43f7u44AjjUhjR99SKg6ZTAgK06tAoxS4HglsFy36eBa/khcdmJ3ubDkydLOn62AnvkayuwfYnpnzzw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.35) smtp.rcpttodomain=marvell.com smtp.mailfrom=nvidia.com; dmarc=pass (p=quarantine sp=none pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=QqTAAYwXQwigIi/k4FYFVIXpI4MtI+tQdOwA2kOIUx0=; b=IHmU1xGfnPRi6b+3hHysRs+8TJfozsGBPHQGbhCptWIbiXFAWd0s64rWMg8FnU/QC29uxiR9rjd2GvNZnErztbj3lpR6UGzJCrfUUFldz4MfhKpiYyqqhuYjshoJhDx+381RHU4roIsN247lyI96ACNvXdkDFh2OSYq1Co0DSWStulysRX8IbCesUeL70RGbtevTfB1/y2EFz1SFu2gG96DGcMrSEGMc9CFwTb40SL8wUE5X2jFgk99RcXGLbcUAuCrSFG5fkXBz8PCyLhtThFITPuGSzYee0XhuCpDEGXX63bejfbGFI9HcMW9tC7UY6ksYAbYc4nVNH+MwAZVWGQ== Received: from DM6PR13CA0015.namprd13.prod.outlook.com (2603:10b6:5:bc::28) by CO6PR12MB5395.namprd12.prod.outlook.com (2603:10b6:303:13a::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4352.26; Thu, 22 Jul 2021 11:04:20 +0000 Received: from DM6NAM11FT066.eop-nam11.prod.protection.outlook.com (2603:10b6:5:bc:cafe::2f) by DM6PR13CA0015.outlook.office365.com (2603:10b6:5:bc::28) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4373.7 via Frontend Transport; Thu, 22 Jul 2021 11:04:20 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.35) smtp.mailfrom=nvidia.com; marvell.com; dkim=none (message not signed) header.d=none; marvell.com; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.35 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.35; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.35) by DM6NAM11FT066.mail.protection.outlook.com (10.13.173.179) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4352.24 via Frontend Transport; Thu, 22 Jul 2021 11:04:20 +0000 Received: from HQMAIL107.nvidia.com (172.20.187.13) by HQMAIL111.nvidia.com (172.20.187.18) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Thu, 22 Jul 2021 11:04:18 +0000 Received: from vdi.nvidia.com (172.20.187.5) by mail.nvidia.com (172.20.187.13) with Microsoft SMTP Server id 15.0.1497.2 via Frontend Transport; Thu, 22 Jul 2021 11:04:13 +0000 From: Boris Pismenny To: , , , , , , , , , , CC: , , , , , , Boris Pismenny , Ben Ben-Ishay , Or Gerlitz , Yoray Zack Subject: [PATCH v5 net-next 01/36] net: Introduce direct data placement tcp offload Date: Thu, 22 Jul 2021 14:02:50 +0300 Message-ID: <20210722110325.371-2-borisp@nvidia.com> X-Mailer: git-send-email 2.24.1 In-Reply-To: <20210722110325.371-1-borisp@nvidia.com> References: <20210722110325.371-1-borisp@nvidia.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 6243351e-d30c-4fa4-7e45-08d94d00749e X-MS-TrafficTypeDiagnostic: CO6PR12MB5395: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:901; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: mvlE8K6Hzw6T0wfrBhMJ/wkwSMfkoD7sjYKI6qCe4VbpKq15Y/zwOvA7D31ZDMtDBMSIgc96ptmivVgqgjo1+7VHtu6chCKKbcbc0X8ODwU32imXr3tvr7t6WIF0fhs+zQ4LsL0jDw7q6da/oG4QLD6nTJR3nl4OOIH9gR1udC8TWvwwTy+exYssxmmeU1L/AfG9QQM60Hs/O6ZFSJPLsbB1z8A3LIjGOp3Byx+LdA/I5On/HdC6wWBkrkzwqYdFTOtTUVuTelH1FD+z8rAx5IthfXXnf66jfkdqpjlMJGAaxdVtfTXquRhXSrFi/1sHGFykjLiU6t4q16vcxIHR8bgA0TFb2s7hGFLciK6X58zyypDuNzUpXFLjCF2Jp+KvRcKMqV2IHMTNc8AbmN572M6LGT+Dj8R9D6mZWSqLjh9RidZIpCczgg92XKfcvsG7FDGtK1q/CeiP6cOS5555QwYVV4Wt9zj71N1gcF+AA+Y31vEvbMzKrID4MfELCzmAJqh4m+g/4WBtvdfQINf8mcJ8dC+SLM06Tj6lJpcuC/Cszn5VKjM/SrgghNBm9CxqlUSrbSAKSdcym80DnyPMF3dZ5UcMbIlqXnnpKqSIYYx33CgaEq8TUl+zQ9yAJ2nMr82wINOpySEyoCRsEoaXUcg9GWdrzeLHNuvgXx26dQR6rPedyvvIApYeow7IfQfMKohp/+yiltmA0RxWFr3SbK4cwHjjKpImpIDvisIVhtw= X-Forefront-Antispam-Report: CIP:216.228.112.35; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:schybrid04.nvidia.com; CAT:NONE; SFS:(4636009)(376002)(346002)(39860400002)(136003)(396003)(46966006)(36840700001)(7696005)(70586007)(6666004)(2616005)(8936002)(478600001)(82310400003)(7416002)(107886003)(7636003)(8676002)(316002)(47076005)(1076003)(186003)(356005)(70206006)(26005)(36906005)(2906002)(336012)(5660300002)(36860700001)(54906003)(426003)(82740400003)(30864003)(4326008)(921005)(83380400001)(110136005)(86362001)(36756003); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Jul 2021 11:04:20.0459 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 6243351e-d30c-4fa4-7e45-08d94d00749e X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.112.35]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DM6NAM11FT066.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CO6PR12MB5395 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Boris Pismenny This commit introduces direct data placement offload for TCP. This capability is accompanied by new net_device operations that configure hardware contexts. There is a context per socket, and a context per DDP opreation. Additionally, a resynchronization routine is used to assist hardware handle TCP OOO, and continue the offload. Furthermore, we let the offloading driver advertise what is the max hw sectors/segments. Using this interface, the NIC hardware will scatter TCP payload directly to the BIO pages according to the command_id. To maintain the correctness of the network stack, the driver is expected to construct SKBs that point to the BIO pages. The SKB passed to the network stack from the driver represents data as it is on the wire, while it is pointing directly to data in destination buffers. As a result, data from page frags should not be copied out to the linear part. To avoid needless copies, such as when using skb_condense, we mark the skb->ddp_crc bit. This bit will be used to indicate both ddp and crc offload (next patch in series). A follow-up patch will use this interface for DDP in NVMe-TCP. Signed-off-by: Boris Pismenny Signed-off-by: Ben Ben-Ishay Signed-off-by: Or Gerlitz Signed-off-by: Yoray Zack --- include/linux/netdev_features.h | 3 +- include/linux/netdevice.h | 5 ++ include/linux/skbuff.h | 4 + include/net/inet_connection_sock.h | 4 + include/net/ulp_ddp.h | 136 +++++++++++++++++++++++++++++ net/Kconfig | 10 +++ net/core/skbuff.c | 8 +- net/ethtool/common.c | 1 + net/ipv4/tcp_input.c | 8 ++ net/ipv4/tcp_ipv4.c | 3 + net/ipv4/tcp_offload.c | 3 + 11 files changed, 183 insertions(+), 2 deletions(-) create mode 100644 include/net/ulp_ddp.h diff --git a/include/linux/netdev_features.h b/include/linux/netdev_features.h index 2c6b9e416225..d9bd6ea26fc8 100644 --- a/include/linux/netdev_features.h +++ b/include/linux/netdev_features.h @@ -14,7 +14,7 @@ typedef u64 netdev_features_t; enum { NETIF_F_SG_BIT, /* Scatter/gather IO. */ NETIF_F_IP_CSUM_BIT, /* Can checksum TCP/UDP over IPv4. */ - __UNUSED_NETIF_F_1, + NETIF_F_HW_ULP_DDP_BIT, /* ULP direct data placement offload */ NETIF_F_HW_CSUM_BIT, /* Can checksum all the packets. */ NETIF_F_IPV6_CSUM_BIT, /* Can checksum TCP/UDP over IPV6 */ NETIF_F_HIGHDMA_BIT, /* Can DMA to high memory. */ @@ -168,6 +168,7 @@ enum { #define NETIF_F_HW_HSR_TAG_RM __NETIF_F(HW_HSR_TAG_RM) #define NETIF_F_HW_HSR_FWD __NETIF_F(HW_HSR_FWD) #define NETIF_F_HW_HSR_DUP __NETIF_F(HW_HSR_DUP) +#define NETIF_F_HW_ULP_DDP __NETIF_F(HW_ULP_DDP) /* Finds the next feature with the highest number of the range of start till 0. */ diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h index eaf5bb008aa9..cba92c2dd9c0 100644 --- a/include/linux/netdevice.h +++ b/include/linux/netdevice.h @@ -1005,6 +1005,7 @@ struct dev_ifalias { struct devlink; struct tlsdev_ops; +struct ulp_ddp_dev_ops; struct netdev_name_node { struct hlist_node hlist; @@ -2024,6 +2025,10 @@ struct net_device { const struct tlsdev_ops *tlsdev_ops; #endif +#if IS_ENABLED(CONFIG_ULP_DDP) + const struct ulp_ddp_dev_ops *ulp_ddp_ops; +#endif + const struct header_ops *header_ops; unsigned char operstate; diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h index b2db9cd9a73f..d323ecd37448 100644 --- a/include/linux/skbuff.h +++ b/include/linux/skbuff.h @@ -689,6 +689,7 @@ typedef unsigned char *sk_buff_data_t; * CHECKSUM_UNNECESSARY (max 3) * @dst_pending_confirm: need to confirm neighbour * @decrypted: Decrypted SKB + * @ddp_crc: DDP or CRC offloaded * @napi_id: id of the NAPI struct this skb came from * @sender_cpu: (aka @napi_id) source CPU in XPS * @secmark: security marking @@ -870,6 +871,9 @@ struct sk_buff { #ifdef CONFIG_TLS_DEVICE __u8 decrypted:1; #endif +#ifdef CONFIG_ULP_DDP + __u8 ddp_crc:1; +#endif #ifdef CONFIG_NET_SCHED __u16 tc_index; /* traffic control index */ diff --git a/include/net/inet_connection_sock.h b/include/net/inet_connection_sock.h index b06c2d02ec84..66801ea72fb4 100644 --- a/include/net/inet_connection_sock.h +++ b/include/net/inet_connection_sock.h @@ -66,6 +66,8 @@ struct inet_connection_sock_af_ops { * @icsk_ulp_ops Pluggable ULP control hook * @icsk_ulp_data ULP private data * @icsk_clean_acked Clean acked data hook + * @icsk_ulp_ddp_ops Pluggable ULP direct data placement control hook + * @icsk_ulp_ddp_data ULP direct data placement private data * @icsk_listen_portaddr_node hash to the portaddr listener hashtable * @icsk_ca_state: Congestion control state * @icsk_retransmits: Number of unrecovered [RTO] timeouts @@ -96,6 +98,8 @@ struct inet_connection_sock { const struct tcp_ulp_ops *icsk_ulp_ops; void __rcu *icsk_ulp_data; void (*icsk_clean_acked)(struct sock *sk, u32 acked_seq); + const struct ulp_ddp_ulp_ops *icsk_ulp_ddp_ops; + void __rcu *icsk_ulp_ddp_data; struct hlist_node icsk_listen_portaddr_node; unsigned int (*icsk_sync_mss)(struct sock *sk, u32 pmtu); __u8 icsk_ca_state:5, diff --git a/include/net/ulp_ddp.h b/include/net/ulp_ddp.h new file mode 100644 index 000000000000..1a0b464ff40b --- /dev/null +++ b/include/net/ulp_ddp.h @@ -0,0 +1,136 @@ +/* SPDX-License-Identifier: GPL-2.0 + * + * ulp_ddp.h + * Author: Boris Pismenny + * Copyright (C) 2021 Mellanox Technologies. + */ +#ifndef _ULP_DDP_H +#define _ULP_DDP_H + +#include +#include +#include + +/* limits returned by the offload driver, zero means don't care */ +struct ulp_ddp_limits { + int max_ddp_sgl_len; +}; + +enum ulp_ddp_type { + ULP_DDP_NVME = 1, +}; + +/** + * struct ulp_ddp_config - Generic ulp ddp configuration: tcp ddp IO queue + * config implementations must use this as the first member. + * Add new instances of ulp_ddp_config below (nvme-tcp, etc.). + */ +struct ulp_ddp_config { + enum ulp_ddp_type type; + unsigned char buf[]; +}; + +/** + * struct nvme_tcp_ddp_config - nvme tcp ddp configuration for an IO queue + * + * @pfv: pdu version (e.g., NVME_TCP_PFV_1_0) + * @cpda: controller pdu data alignmend (dwords, 0's based) + * @dgst: digest types enabled. + * The netdev will offload crc if ddp_crc is supported. + * @queue_size: number of nvme-tcp IO queue elements + * @queue_id: queue identifier + * @cpu_io: cpu core running the IO thread for this queue + */ +struct nvme_tcp_ddp_config { + struct ulp_ddp_config cfg; + + u16 pfv; + u8 cpda; + u8 dgst; + int queue_size; + int queue_id; + int io_cpu; +}; + +/** + * struct ulp_ddp_io - ulp ddp configuration for an IO request. + * + * @command_id: identifier on the wire associated with these buffers + * @nents: number of entries in the sg_table + * @sg_table: describing the buffers for this IO request + * @first_sgl: first SGL in sg_table + */ +struct ulp_ddp_io { + u32 command_id; + int nents; + struct sg_table sg_table; + struct scatterlist first_sgl[SG_CHUNK_SIZE]; +}; + +/* struct ulp_ddp_dev_ops - operations used by an upper layer protocol to configure ddp offload + * + * @ulp_ddp_limits: limit the number of scatter gather entries per IO. + * the device driver can use this to limit the resources allocated per queue. + * @ulp_ddp_sk_add: add offload for the queue represennted by the socket+config pair. + * this function is used to configure either copy, crc or both offloads. + * @ulp_ddp_sk_del: remove offload from the socket, and release any device related resources. + * @ulp_ddp_setup: request copy offload for buffers associated with a command_id in ulp_ddp_io. + * @ulp_ddp_teardown: release offload resources association between buffers and command_id in + * ulp_ddp_io. + * @ulp_ddp_resync: respond to the driver's resync_request. Called only if resync is successful. + */ +struct ulp_ddp_dev_ops { + int (*ulp_ddp_limits)(struct net_device *netdev, + struct ulp_ddp_limits *limits); + int (*ulp_ddp_sk_add)(struct net_device *netdev, + struct sock *sk, + struct ulp_ddp_config *config); + void (*ulp_ddp_sk_del)(struct net_device *netdev, + struct sock *sk); + int (*ulp_ddp_setup)(struct net_device *netdev, + struct sock *sk, + struct ulp_ddp_io *io); + int (*ulp_ddp_teardown)(struct net_device *netdev, + struct sock *sk, + struct ulp_ddp_io *io, + void *ddp_ctx); + void (*ulp_ddp_resync)(struct net_device *netdev, + struct sock *sk, u32 seq); +}; + +#define ULP_DDP_RESYNC_REQ BIT(0) + +/** + * struct ulp_ddp_ulp_ops - Interface to register uppper layer Direct Data Placement (DDP) TCP offload + */ +struct ulp_ddp_ulp_ops { + /* NIC requests ulp to indicate if @seq is the start of a message */ + bool (*resync_request)(struct sock *sk, u32 seq, u32 flags); + /* NIC driver informs the ulp that ddp teardown is done - used for async completions*/ + void (*ddp_teardown_done)(void *ddp_ctx); +}; + +/** + * struct ulp_ddp_ctx - Generic ulp ddp context: device driver per queue contexts must + * use this as the first member. + */ +struct ulp_ddp_ctx { + enum ulp_ddp_type type; + unsigned char buf[]; +}; + +static inline struct ulp_ddp_ctx *ulp_ddp_get_ctx(const struct sock *sk) +{ + struct inet_connection_sock *icsk = inet_csk(sk); + + return (__force struct ulp_ddp_ctx *)icsk->icsk_ulp_ddp_data; +} + +static inline void ulp_ddp_set_ctx(struct sock *sk, void *ctx) +{ + struct inet_connection_sock *icsk = inet_csk(sk); + + rcu_assign_pointer(icsk->icsk_ulp_ddp_data, ctx); +} + +#endif //_ULP_DDP_H diff --git a/net/Kconfig b/net/Kconfig index c7392c449b25..b6f0ccbea1e3 100644 --- a/net/Kconfig +++ b/net/Kconfig @@ -454,4 +454,14 @@ config ETHTOOL_NETLINK netlink. It provides better extensibility and some new features, e.g. notification messages. +config ULP_DDP + bool "ULP direct data placement offload" + default n + help + Direct Data Placement (DDP) offload enables ULP, such as + NVMe-TCP/iSCSI, to request the NIC to place ULP payload data + of a command response directly into kernel pages while + calculate/verify the data digest on ULP PDU as they go through + the NIC. Thus avoiding the costly per-byte overhead. + endif # if NET diff --git a/net/core/skbuff.c b/net/core/skbuff.c index 12aabcda6db2..20add6c3f2e6 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -71,6 +71,7 @@ #include #include #include +#include #include #include @@ -6295,9 +6296,14 @@ EXPORT_SYMBOL(pskb_extract); */ void skb_condense(struct sk_buff *skb) { + bool is_ddp = false; + +#ifdef CONFIG_ULP_DDP + is_ddp = skb->ddp_crc; +#endif if (skb->data_len) { if (skb->data_len > skb->end - skb->tail || - skb_cloned(skb)) + skb_cloned(skb) || is_ddp) return; /* Nice, we can free page frag(s) right now */ diff --git a/net/ethtool/common.c b/net/ethtool/common.c index f9dcbad84788..d545d1525800 100644 --- a/net/ethtool/common.c +++ b/net/ethtool/common.c @@ -73,6 +73,7 @@ const char netdev_features_strings[NETDEV_FEATURE_COUNT][ETH_GSTRING_LEN] = { [NETIF_F_HW_HSR_TAG_RM_BIT] = "hsr-tag-rm-offload", [NETIF_F_HW_HSR_FWD_BIT] = "hsr-fwd-offload", [NETIF_F_HW_HSR_DUP_BIT] = "hsr-dup-offload", + [NETIF_F_HW_ULP_DDP_BIT] = "ulp-ddp-offload", }; const char diff --git a/net/ipv4/tcp_input.c b/net/ipv4/tcp_input.c index e6ca5a1f3b59..4a7160bba09b 100644 --- a/net/ipv4/tcp_input.c +++ b/net/ipv4/tcp_input.c @@ -5149,6 +5149,9 @@ tcp_collapse(struct sock *sk, struct sk_buff_head *list, struct rb_root *root, memcpy(nskb->cb, skb->cb, sizeof(skb->cb)); #ifdef CONFIG_TLS_DEVICE nskb->decrypted = skb->decrypted; +#endif +#ifdef CONFIG_ULP_DDP + nskb->ddp_crc = skb->ddp_crc; #endif TCP_SKB_CB(nskb)->seq = TCP_SKB_CB(nskb)->end_seq = start; if (list) @@ -5182,6 +5185,11 @@ tcp_collapse(struct sock *sk, struct sk_buff_head *list, struct rb_root *root, #ifdef CONFIG_TLS_DEVICE if (skb->decrypted != nskb->decrypted) goto end; +#endif +#ifdef CONFIG_ULP_DDP + + if (skb->ddp_crc != nskb->ddp_crc) + goto end; #endif } } diff --git a/net/ipv4/tcp_ipv4.c b/net/ipv4/tcp_ipv4.c index e66ad6bfe808..3d9849a39b82 100644 --- a/net/ipv4/tcp_ipv4.c +++ b/net/ipv4/tcp_ipv4.c @@ -1830,6 +1830,9 @@ bool tcp_add_backlog(struct sock *sk, struct sk_buff *skb) TCP_SKB_CB(skb)->tcp_flags) & (TCPHDR_ECE | TCPHDR_CWR)) || #ifdef CONFIG_TLS_DEVICE tail->decrypted != skb->decrypted || +#endif +#ifdef CONFIG_ULP_DDP + tail->ddp_crc != skb->ddp_crc || #endif thtail->doff != th->doff || memcmp(thtail + 1, th + 1, hdrlen - sizeof(*th))) diff --git a/net/ipv4/tcp_offload.c b/net/ipv4/tcp_offload.c index e09147ac9a99..96e8228d2b96 100644 --- a/net/ipv4/tcp_offload.c +++ b/net/ipv4/tcp_offload.c @@ -262,6 +262,9 @@ struct sk_buff *tcp_gro_receive(struct list_head *head, struct sk_buff *skb) #ifdef CONFIG_TLS_DEVICE flush |= p->decrypted ^ skb->decrypted; #endif +#ifdef CONFIG_ULP_DDP + flush |= p->ddp_crc ^ skb->ddp_crc; +#endif if (flush || skb_gro_receive(p, skb)) { mss = 1; From patchwork Thu Jul 22 11:02:51 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Boris Pismenny X-Patchwork-Id: 484325 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-20.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER, INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9CD13C6377D for ; Thu, 22 Jul 2021 11:04:39 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 78BFB6121E for ; Thu, 22 Jul 2021 11:04:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231661AbhGVKYD (ORCPT ); Thu, 22 Jul 2021 06:24:03 -0400 Received: from mail-bn8nam11on2084.outbound.protection.outlook.com ([40.107.236.84]:41889 "EHLO NAM11-BN8-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S231628AbhGVKXv (ORCPT ); Thu, 22 Jul 2021 06:23:51 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=QlDV84A2+LXPEeYJiABOiQVlrtSY6D26/4s4o1QAI5hNxiD5fAf2qzwpmSmraMMp1S7xQScEsa7SWOI9xRnFNQ66P3aXPv4LhNjfDLH3WbD8djbH1+/uUvgdIG8CDlVjyT+OeHYiKjkg+g0y+PV6wet+/CynCZ1VZmD0/dHDPJT1DTMqQST69kT2QjdMYN6hZd5dtIUIv5sE4AlVooEzSF09L5dWPv9vstjdkSzy7x9tXlzXrWatmAOLSQDEZCXxPSpC+EEJnTAtywdDVxDMnzLDvLeHX+MW/PyiXaW7kB5V5j63CikzM9bKSYkJLCaF6afEiyv/s8V0FCFloKxElQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=xU5l6aDtsceNgoUrW3pxgrtmqwiI7usvRwB8T+szLtQ=; b=QehK6VS5tTLvEHVHYnaKzslYi6CHu2U2u4YgfZBMyHTfbeMOULRnSEbrMTkeqKzcgiy6t8a4jrc5ymX35SOb0Bhq4iAm9ZcN46p2a1FC1bpAQ9BFzaFizrbbaqOtHT5G1qxiiIRHvJfSHfUQzZyCg3dH4IHqFBQkXVHJVsjKe7fUHLMHjxaD6xxkgrQlSLawTkfzGZPXJAok2Mgzl6hJCg4RR9IDCztfdMSnnCAtuFD5f1dKbvjRfn4NxRWvOK+Xjhm7b/OwsGu4px2mV0O5aVt27nSih4X2885Uecl0tG8XCIi4ttf3ZDzWDSMA4SLHahPZ1Rg5ragq8M2xb4dzEQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.34) smtp.rcpttodomain=google.com smtp.mailfrom=nvidia.com; dmarc=pass (p=quarantine sp=none pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=xU5l6aDtsceNgoUrW3pxgrtmqwiI7usvRwB8T+szLtQ=; b=DiI8Xdf+XH7R51hS+OBIMmNHuZN1uThJJOk+cNbCwK1d/s7G0bXLTdskB4h6Kw8gXzlX4Y/2bIUW/JZ4+9ur5h2CoFBSJUzzSfwBh1zktyOL2zPtiG/mdqhPGmmQxQncIMwpJsGLJ+bxx/E5hiCFurcsLX3i00xZLdSi3Kj5ERiWZGVVF6ZK3BDNCCuc48CDJql8PmjpYRbZGus2MiW04uG52QDMM15BiM98jwwhnxSaOt6d2RIB9TrBlbsbDWkH5hhjZLytZxDE0udtfrzj7tjg1fiO9Qk3uHVe5B1JI2Uqs5oGM7ZXoTVSjZtQrXAeIFb4KH1a5K9azjuhtHdPJQ== Received: from BN0PR04CA0076.namprd04.prod.outlook.com (2603:10b6:408:ea::21) by CY4PR12MB1575.namprd12.prod.outlook.com (2603:10b6:910:f::23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4331.29; Thu, 22 Jul 2021 11:04:25 +0000 Received: from BN8NAM11FT056.eop-nam11.prod.protection.outlook.com (2603:10b6:408:ea:cafe::77) by BN0PR04CA0076.outlook.office365.com (2603:10b6:408:ea::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4352.25 via Frontend Transport; Thu, 22 Jul 2021 11:04:25 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.34) smtp.mailfrom=nvidia.com; google.com; dkim=none (message not signed) header.d=none; google.com; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.34 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.34; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.34) by BN8NAM11FT056.mail.protection.outlook.com (10.13.177.26) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4352.24 via Frontend Transport; Thu, 22 Jul 2021 11:04:24 +0000 Received: from HQMAIL105.nvidia.com (172.20.187.12) by HQMAIL107.nvidia.com (172.20.187.13) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Thu, 22 Jul 2021 11:04:24 +0000 Received: from HQMAIL107.nvidia.com (172.20.187.13) by HQMAIL105.nvidia.com (172.20.187.12) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Thu, 22 Jul 2021 11:04:24 +0000 Received: from vdi.nvidia.com (172.20.187.5) by mail.nvidia.com (172.20.187.13) with Microsoft SMTP Server id 15.0.1497.2 via Frontend Transport; Thu, 22 Jul 2021 11:04:19 +0000 From: Boris Pismenny To: , , , , , , , , , , CC: , , , , , , Boris Pismenny , Ben Ben-Ishay , Or Gerlitz , Yoray Zack Subject: [PATCH v5 net-next 02/36] iov_iter: DDP copy to iter/pages Date: Thu, 22 Jul 2021 14:02:51 +0300 Message-ID: <20210722110325.371-3-borisp@nvidia.com> X-Mailer: git-send-email 2.24.1 In-Reply-To: <20210722110325.371-1-borisp@nvidia.com> References: <20210722110325.371-1-borisp@nvidia.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: cc1bda8f-d512-4983-c86f-08d94d007797 X-MS-TrafficTypeDiagnostic: CY4PR12MB1575: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:530; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: AFklZ36uzu0+SigaEwzgPzh1c2C/vLV63T1wxRTGmA70w59x9tevJ7SP6Hi9rAisyzopfRKp9MmCQguKmKbdK2F5QE17X0i52V8p1iqxhk55p0Pz2ulL3eokSgR6peP0LgqDctw5z8tcbjjVDsfF6KUtBs6wg0oxWlQOKe1dMae8kQTU7BeDGzjyBXVLLP+EAavUQqmb7376prcyrtwvAjaGqCUnyP4uouYwU8vgms2lHpVlrFg4lN7qW+PPZ/S+aniK8Q4OwqL1Iv/KbWiTMV+NZD7fXeh9mOM0szjAYyOrYge6K9hakjraMeNshVwRVvXf2qRrmHM7pnmGiX7Mf51hsuBLPSE9YkLvCcbrT7Ol0fZ1gsNMCiuGYaocigWIKZHqeSGNZ3qwJmWQNTvVaTZ58P+pDWq7Wl1k4Iv3GLwFVY4t9zh21+n2a9jZECNfzAeHp2TnN3WMtOwl8mObxHcBi4vrjF34cHyyeFKUxBhlqNGlKT4JPa2cgejnQ0fReF6soap4hIA738jrIKhD0S/aPWI+8WvZ032PhHC5VJEK9pbSkecsAhlZKmOaGjZGsJddBg+BtzcOsAdSG63jBm3KQGcIm70Cc87Bc6m8Rj7WiZHxtPVEtAcf0BUFSokDVStPmWZ3wGMnTbdce/ZiH3PancdPgmAtXVIEmBP/A4MPyBb+1D5BhUgJQY6Jw9yYj9KpEK1hY+j1x/Iyi7riqDrMC5rA5gUvZPVtnoPfDG8= X-Forefront-Antispam-Report: CIP:216.228.112.34; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:schybrid03.nvidia.com; CAT:NONE; SFS:(4636009)(346002)(396003)(376002)(136003)(39860400002)(36840700001)(46966006)(5660300002)(186003)(4326008)(1076003)(36906005)(54906003)(356005)(7636003)(316002)(82310400003)(8936002)(36860700001)(26005)(426003)(8676002)(6666004)(2906002)(7696005)(336012)(110136005)(478600001)(83380400001)(2616005)(36756003)(107886003)(70586007)(921005)(47076005)(86362001)(70206006)(82740400003)(7416002); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Jul 2021 11:04:24.9721 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: cc1bda8f-d512-4983-c86f-08d94d007797 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.112.34]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BN8NAM11FT056.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY4PR12MB1575 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Boris Pismenny When using direct data placement (DDP) the NIC writes some of the payload directly to the destination buffer, and constructs SKBs such that they point to this data. To skip copies when SKB data already resides in the destination we use the newly introduced routines in this commit, which check if (src == dst), and skip the copy when that's true. As the current user for these routines is in the block layer (nvme-tcp), then we only apply the change for bio_vec. Other routines use the normal methods for copying. Signed-off-by: Boris Pismenny Signed-off-by: Ben Ben-Ishay Signed-off-by: Or Gerlitz Signed-off-by: Yoray Zack --- include/linux/uio.h | 17 ++++++++++++++ lib/iov_iter.c | 55 +++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 72 insertions(+) diff --git a/include/linux/uio.h b/include/linux/uio.h index d3ec87706d75..a61fdb369e0e 100644 --- a/include/linux/uio.h +++ b/include/linux/uio.h @@ -131,6 +131,9 @@ size_t copy_page_from_iter(struct page *page, size_t offset, size_t bytes, struct iov_iter *i); size_t _copy_to_iter(const void *addr, size_t bytes, struct iov_iter *i); +#ifdef CONFIG_ULP_DDP +size_t _ddp_copy_to_iter(const void *addr, size_t bytes, struct iov_iter *i); +#endif size_t _copy_from_iter(void *addr, size_t bytes, struct iov_iter *i); bool _copy_from_iter_full(void *addr, size_t bytes, struct iov_iter *i); size_t _copy_from_iter_nocache(void *addr, size_t bytes, struct iov_iter *i); @@ -145,6 +148,16 @@ size_t copy_to_iter(const void *addr, size_t bytes, struct iov_iter *i) return _copy_to_iter(addr, bytes, i); } +#ifdef CONFIG_ULP_DDP +static __always_inline __must_check +size_t ddp_copy_to_iter(const void *addr, size_t bytes, struct iov_iter *i) +{ + if (unlikely(!check_copy_size(addr, bytes, true))) + return 0; + return _ddp_copy_to_iter(addr, bytes, i); +} +#endif + static __always_inline __must_check size_t copy_from_iter(void *addr, size_t bytes, struct iov_iter *i) { @@ -281,6 +294,10 @@ size_t csum_and_copy_from_iter(void *addr, size_t bytes, __wsum *csum, struct io bool csum_and_copy_from_iter_full(void *addr, size_t bytes, __wsum *csum, struct iov_iter *i); size_t hash_and_copy_to_iter(const void *addr, size_t bytes, void *hashp, struct iov_iter *i); +#ifdef CONFIG_ULP_DDP +size_t ddp_hash_and_copy_to_iter(const void *addr, size_t bytes, void *hashp, + struct iov_iter *i); +#endif struct iovec *iovec_from_user(const struct iovec __user *uvector, unsigned long nr_segs, unsigned long fast_segs, diff --git a/lib/iov_iter.c b/lib/iov_iter.c index c701b7a187f2..2e9be46a9b56 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -508,6 +508,18 @@ void iov_iter_init(struct iov_iter *i, unsigned int direction, } EXPORT_SYMBOL(iov_iter_init); +#ifdef CONFIG_ULP_DDP +static void ddp_memcpy_to_page(struct page *page, size_t offset, const char *from, size_t len) +{ + char *to = kmap_atomic(page); + + if (to + offset != from) + memcpy(to + offset, from, len); + + kunmap_atomic(to); +} +#endif + static inline bool allocated(struct pipe_buffer *buf) { return buf->ops == &default_pipe_buf_ops; @@ -648,6 +660,28 @@ static size_t csum_and_copy_to_pipe_iter(const void *addr, size_t bytes, return bytes; } +#ifdef CONFIG_ULP_DDP +size_t _ddp_copy_to_iter(const void *addr, size_t bytes, struct iov_iter *i) +{ + const char *from = addr; + if (unlikely(iov_iter_is_pipe(i))) + return copy_pipe_to_iter(addr, bytes, i); + if (iter_is_iovec(i)) + might_fault(); + iterate_and_advance(i, bytes, v, + copyout(v.iov_base, (from += v.iov_len) - v.iov_len, v.iov_len), + ddp_memcpy_to_page(v.bv_page, v.bv_offset, + (from += v.bv_len) - v.bv_len, v.bv_len), + memcpy(v.iov_base, (from += v.iov_len) - v.iov_len, v.iov_len), + ddp_memcpy_to_page(v.bv_page, v.bv_offset, + (from += v.bv_len) - v.bv_len, v.bv_len) + ) + + return bytes; +} +EXPORT_SYMBOL(_ddp_copy_to_iter); +#endif + size_t _copy_to_iter(const void *addr, size_t bytes, struct iov_iter *i) { const char *from = addr; @@ -1818,6 +1852,27 @@ size_t csum_and_copy_to_iter(const void *addr, size_t bytes, void *_csstate, } EXPORT_SYMBOL(csum_and_copy_to_iter); +#ifdef CONFIG_ULP_DDP +size_t ddp_hash_and_copy_to_iter(const void *addr, size_t bytes, void *hashp, + struct iov_iter *i) +{ +#ifdef CONFIG_CRYPTO_HASH + struct ahash_request *hash = hashp; + struct scatterlist sg; + size_t copied; + + copied = ddp_copy_to_iter(addr, bytes, i); + sg_init_one(&sg, addr, copied); + ahash_request_set_crypt(hash, &sg, NULL, copied); + crypto_ahash_update(hash); + return copied; +#else + return 0; +#endif +} +EXPORT_SYMBOL(ddp_hash_and_copy_to_iter); +#endif + size_t hash_and_copy_to_iter(const void *addr, size_t bytes, void *hashp, struct iov_iter *i) { From patchwork Thu Jul 22 11:02:52 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Boris Pismenny X-Patchwork-Id: 485111 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-20.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER, INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3F140C6377D for ; Thu, 22 Jul 2021 11:04:45 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 2201D60725 for ; Thu, 22 Jul 2021 11:04:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231678AbhGVKYE (ORCPT ); Thu, 22 Jul 2021 06:24:04 -0400 Received: from mail-bn1nam07on2046.outbound.protection.outlook.com ([40.107.212.46]:21252 "EHLO NAM02-BN1-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S231629AbhGVKX4 (ORCPT ); Thu, 22 Jul 2021 06:23:56 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Gz/OzZnCv4mTj2zngNDawBtOVY/vjMdQBXkB/EnngQyQ8571PHlzDdZSUovsiLGe0wJCHhxQGorjYer4JqPcaCvuTQU3oyGJ6Czrdc/2023OkujoajRO/j/lFyAc+2QhSMK4yWyKevHqyku81izNgtvx32M2CObqH0izE7XD3JzguVOIrU91bCQ/UVhK7b/jAbptszjso2eWYpxkui3lAQzeaw/JTOrpaFLRc8cA21bF5HUyRfoRpqkLq6Uw7IDckWeBfpETdgYOS/SPAEVTl6ecXs8Md2ayNBLUeQ6IBa+pQJXMw4OAcSolARdV/eBR2C357aw6B++vmiw0cFcnqg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=/aMa/LbdVVpeKmbblabEig4UwdwuOWkRhu0YaUZoYC8=; b=H4Tz94q09hMh5AHUIyZ1xuFyPfPt265Nv8azwFSbT+o4BJkQ/mdMOWS9hbdZU+sj1Y3dzjMngrnjNQR08o/k9cTuZiWdo7G1ys65vNHpbpYc/7QzCcq7VQeeM7DKI3NyIblJ/RHK4XK45Hko6Kb+eyWT2HbIMt+vM0DCS5Y7rhrsOcp/xuMiMxu/Hc/kPSDYMgev5URsNW4aMZ64cCvjDLb8a6N7Q/gUOZmJCQq52ZRsx/VPvRIy+4H9x5IgiRIWzzsdCPPzfX9C+1N8lB2N625fkzSbWakto2YBAemp0i1oKT0+qctY+w55ksZ3oT9rdqyYhKELakc1sHAtXZMmGA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.36) smtp.rcpttodomain=marvell.com smtp.mailfrom=nvidia.com; dmarc=pass (p=quarantine sp=none pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=/aMa/LbdVVpeKmbblabEig4UwdwuOWkRhu0YaUZoYC8=; b=ZOkajTKqJWB2b5c1RaYaF/2hyfUXDcjloRhfdCqZlDOWTZXHhtatP3pEl7BQug64OhtiApySTagyA2Mum+s6e19CvYmiFYqPWjYKvZkCysoWyQMqgsXcn7i2n90X7+FDu+woASdLwqFksXV2jnk1Nkym4FMW7ni1QpL4in5xsHNqARGF8+t3b4AhCapZ6lVh7nqkY+Dz215DMP3NAv1DD9c3JlwwPKbpfbJ2y4V0AA9TTflFYKO2ANQdsSLiGSZW0eUI0etNl4sT3+njm9E0Fe7V9hf+kkL1OgWwvcZEBLpvgjf6zjpu4B+NdNw8F7jytoaelGstiZYIqIbCiifvAA== Received: from BN9PR03CA0320.namprd03.prod.outlook.com (2603:10b6:408:112::25) by MN2PR12MB3486.namprd12.prod.outlook.com (2603:10b6:208:c6::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4352.24; Thu, 22 Jul 2021 11:04:30 +0000 Received: from BN8NAM11FT026.eop-nam11.prod.protection.outlook.com (2603:10b6:408:112:cafe::46) by BN9PR03CA0320.outlook.office365.com (2603:10b6:408:112::25) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4352.25 via Frontend Transport; Thu, 22 Jul 2021 11:04:30 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.36) smtp.mailfrom=nvidia.com; marvell.com; dkim=none (message not signed) header.d=none; marvell.com; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.36 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.36; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.36) by BN8NAM11FT026.mail.protection.outlook.com (10.13.177.51) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4352.24 via Frontend Transport; Thu, 22 Jul 2021 11:04:29 +0000 Received: from HQMAIL107.nvidia.com (172.20.187.13) by HQMAIL101.nvidia.com (172.20.187.10) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Thu, 22 Jul 2021 11:04:28 +0000 Received: from vdi.nvidia.com (172.20.187.5) by mail.nvidia.com (172.20.187.13) with Microsoft SMTP Server id 15.0.1497.2 via Frontend Transport; Thu, 22 Jul 2021 11:04:24 +0000 From: Boris Pismenny To: , , , , , , , , , , CC: , , , , , , Boris Pismenny , Or Gerlitz , Yoray Zack Subject: [PATCH v5 net-next 03/36] net: skb copy(+hash) iterators for DDP offloads Date: Thu, 22 Jul 2021 14:02:52 +0300 Message-ID: <20210722110325.371-4-borisp@nvidia.com> X-Mailer: git-send-email 2.24.1 In-Reply-To: <20210722110325.371-1-borisp@nvidia.com> References: <20210722110325.371-1-borisp@nvidia.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 354ce2b0-fcd5-4de4-136e-08d94d007a65 X-MS-TrafficTypeDiagnostic: MN2PR12MB3486: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:497; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: Ss+cR4gX0w8eeS1jnnzy6ds/5RWKjAXNUjwzH2uPJzUXqh/xvkwP4LRcTwlNq5UyWdhcHZ8yT3xtZ9AwfcIOyS0SL/ryXULLKZoO4An87ENwTa+jY2jjtXBx8M+4PvNzPr+IfgDPFamae4P2jFV4ZzywDI+XY6RuA3KVLGxIoOxCP/qaLgGfT2gOOzGnn7ke53DjaGFqLtvhJHeoBveFahLzzH+Yoion06UkjuklUrQHN+aN/82dfmlavgiTRkUVfZWBrBRjI5xwK0ExAORoDTD28dUXJx9jEpgFUO5y7/s0ix88qBGdJDEh9XjIgvpSwFv/59EpgNin7a7Ce/kZp/aQYqf+fidhEMRps+TNsNSXzFBQm9tq4QE2johSmvzqjPW+VIpefBCkqDh18TG44tJsF6LofHpckcl1PURz9ZSO24LuqnQdJt8dT1a675acGijF9IXWaK+1ytZt8xE+2+ig+TQBpMOCwAFXqhbvugxzg0H4VbD2XhxJNZWJMHueQ5uign3dB7IcvX/JxGiNziWGfc5IzoAn97uLeoqICavSRQAslCiZzvW4TH1DBp3SoFkt9RfqfyYdAaBOoE89sFb2+5AdsWVR7Wt6e5nsLkRM0Gpw/QQGDMfonv4giDsDHq7U+1lrWY6xRPcVckJql9miSVwvcTY1eCBHTS6umppxyPh3qPN7pelDyjoDjPSA+0ExTMd9WUn8/YySI1kvIaNnh6SeENYNS1Ub530wg8c= X-Forefront-Antispam-Report: CIP:216.228.112.36; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:schybrid05.nvidia.com; CAT:NONE; SFS:(4636009)(39860400002)(136003)(346002)(396003)(376002)(46966006)(36840700001)(2906002)(86362001)(83380400001)(36860700001)(186003)(82310400003)(107886003)(6666004)(921005)(47076005)(8676002)(5660300002)(8936002)(70586007)(70206006)(54906003)(316002)(36756003)(426003)(4326008)(110136005)(36906005)(7696005)(356005)(7416002)(336012)(26005)(7636003)(1076003)(2616005)(478600001)(82740400003); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Jul 2021 11:04:29.6809 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 354ce2b0-fcd5-4de4-136e-08d94d007a65 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.112.36]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BN8NAM11FT026.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR12MB3486 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Boris Pismenny This commit introduces new functions to support direct data placement (DDP) NIC offloads that avoid copying data from SKBs. Later patches will use this for nvme-tcp DDP offload. Signed-off-by: Boris Pismenny Signed-off-by: Ben Ben-Ishay Signed-off-by: Or Gerlitz Signed-off-by: Yoray Zack --- include/linux/skbuff.h | 9 ++++++++ net/core/datagram.c | 48 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 57 insertions(+) diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h index d323ecd37448..8c1bfd7081d1 100644 --- a/include/linux/skbuff.h +++ b/include/linux/skbuff.h @@ -3613,6 +3613,10 @@ __poll_t datagram_poll(struct file *file, struct socket *sock, struct poll_table_struct *wait); int skb_copy_datagram_iter(const struct sk_buff *from, int offset, struct iov_iter *to, int size); +#ifdef CONFIG_TCP_DDP +int skb_ddp_copy_datagram_iter(const struct sk_buff *from, int offset, + struct iov_iter *to, int size); +#endif static inline int skb_copy_datagram_msg(const struct sk_buff *from, int offset, struct msghdr *msg, int size) { @@ -3623,6 +3627,11 @@ int skb_copy_and_csum_datagram_msg(struct sk_buff *skb, int hlen, int skb_copy_and_hash_datagram_iter(const struct sk_buff *skb, int offset, struct iov_iter *to, int len, struct ahash_request *hash); +#ifdef CONFIG_TCP_DDP +int skb_ddp_copy_and_hash_datagram_iter(const struct sk_buff *skb, int offset, + struct iov_iter *to, int len, + struct ahash_request *hash); +#endif int skb_copy_datagram_from_iter(struct sk_buff *skb, int offset, struct iov_iter *from, int len); int zerocopy_sg_from_iter(struct sk_buff *skb, struct iov_iter *frm); diff --git a/net/core/datagram.c b/net/core/datagram.c index 15ab9ffb27fe..d346fd5da22c 100644 --- a/net/core/datagram.c +++ b/net/core/datagram.c @@ -495,6 +495,27 @@ static int __skb_datagram_iter(const struct sk_buff *skb, int offset, return 0; } +#ifdef CONFIG_TCP_DDP +/** + * skb_ddp_copy_and_hash_datagram_iter - Copies datagrams from skb frags to + * an iterator and update a hash. If the iterator and skb frag point to the + * same page and offset, then the copy is skipped. + * @skb: buffer to copy + * @offset: offset in the buffer to start copying from + * @to: iovec iterator to copy to + * @len: amount of data to copy from buffer to iovec + * @hash: hash request to update + */ +int skb_ddp_copy_and_hash_datagram_iter(const struct sk_buff *skb, int offset, + struct iov_iter *to, int len, + struct ahash_request *hash) +{ + return __skb_datagram_iter(skb, offset, to, len, true, + ddp_hash_and_copy_to_iter, hash); +} +EXPORT_SYMBOL(skb_ddp_copy_and_hash_datagram_iter); +#endif + /** * skb_copy_and_hash_datagram_iter - Copy datagram to an iovec iterator * and update a hash. @@ -513,6 +534,33 @@ int skb_copy_and_hash_datagram_iter(const struct sk_buff *skb, int offset, } EXPORT_SYMBOL(skb_copy_and_hash_datagram_iter); +#ifdef CONFIG_TCP_DDP +static size_t simple_ddp_copy_to_iter(const void *addr, size_t bytes, + void *data __always_unused, + struct iov_iter *i) +{ + return ddp_copy_to_iter(addr, bytes, i); +} + +/** + * skb_ddp_copy_datagram_iter - Copies datagrams from skb frags to an + * iterator. If the iterator and skb frag point to the same page and + * offset, then the copy is skipped. + * @skb: buffer to copy + * @offset: offset in the buffer to start copying from + * @to: iovec iterator to copy to + * @len: amount of data to copy from buffer to iovec + */ +int skb_ddp_copy_datagram_iter(const struct sk_buff *skb, int offset, + struct iov_iter *to, int len) +{ + trace_skb_copy_datagram_iovec(skb, len); + return __skb_datagram_iter(skb, offset, to, len, false, + simple_ddp_copy_to_iter, NULL); +} +EXPORT_SYMBOL(skb_ddp_copy_datagram_iter); +#endif + static size_t simple_copy_to_iter(const void *addr, size_t bytes, void *data __always_unused, struct iov_iter *i) { From patchwork Thu Jul 22 11:02:53 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Boris Pismenny X-Patchwork-Id: 484324 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-20.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER, INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6B735C63793 for ; Thu, 22 Jul 2021 11:04:49 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 51C9D60725 for ; Thu, 22 Jul 2021 11:04:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231648AbhGVKYN (ORCPT ); Thu, 22 Jul 2021 06:24:13 -0400 Received: from mail-dm6nam12on2046.outbound.protection.outlook.com ([40.107.243.46]:41953 "EHLO NAM12-DM6-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S231513AbhGVKYA (ORCPT ); Thu, 22 Jul 2021 06:24:00 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=intglNnk8+whoDQQoWXRQ//8uDfiEnRChDPtFgG5+Ar4y24Bp5NNtjX2NpkxxX5VINyEU3FXh/Gam38qlI2I9d5rkHshZylGvV8mqN/xwae3QWgiwp9QnGvwZHPrdivUUYQBwL48AsSzjCOK1iu7XVeATnj9fALenfo0SAvzhh7yQs/FY1GR29oGofudSyKKXlzeBxUzyGaexZ5gGmqN/varpY7MIDmxRFrBtNmMnREDv7Wg3JSiOWzPWDZKgXnwHw88gikuWO2joEr0VlLhJmX3JBSqWrccoJyrNZGMUcl4AwTTOlHMfOWNirNmjylnT5jAgoK+ByWe8ZYqHxssHg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=0d9jublZvcwGcMthhVaxofy1VrIhWLzT/zvHLc6MzU0=; b=aNOqNos5v1zuTeq1l+o27LuQkgbOUEzqnACR51jxgYCggeGDArYc+B81RtXu+0BzyNgbt6Z+jJJmKtjn3ikqZVs5pqfYK2VMx70nxxXnowXms2xU7dh3OJev8zmX1C8t4YqNw+ghyKMCkDydtj503o6rYRIq1+A4v40TAWG6nbtkcHzH2SOop0K5bWUY+SnPpmXtI5tdcKHrgvO+XJH4VuzT1LJtBrBNu5GXEQxEGC7b1frDIr657qOJNcDpLy/QNsuTh9oDLlAiwGCC1dRIfi1ph22QEKuHbussE25+SBoo5q020xjp9unXImC91xOQ94ZoGIo6lX55hfXDTvm3EA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.35) smtp.rcpttodomain=kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=quarantine sp=none pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=0d9jublZvcwGcMthhVaxofy1VrIhWLzT/zvHLc6MzU0=; b=Gb/JnUeCE/L1+22WMzvP5B7iE4uoUKpzwAJI/J1CRrZ6XcjtNlS/wajOvEhzrmMStqwZC1EuX1V0OnNHio+nvzGVg5OeCkiz+a2NlC4Ax5yKO+u4e0b/9zFfA+gNjBihiFQD60fgruUf/0th4Fo29ZPZc+QCcDEQ9oKolpNkO9AVlJ0Nkj16u133B1e5ME2lfT4oUKzQoi40QPYvFIjpSVApbdIXtL1nU9yanI3mEpsj30ZMQ0xIZWGTxf4qMclanHqw1Kz/g/lJAfBcTafbYlPEODKC3TrbGSx+mKdFxp9Ppn+LzI2IghFfKkrjpy0Wl+YshlkmlA7epLlahD1cpA== Received: from DM5PR13CA0034.namprd13.prod.outlook.com (2603:10b6:3:7b::20) by BY5PR12MB4194.namprd12.prod.outlook.com (2603:10b6:a03:210::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4352.26; Thu, 22 Jul 2021 11:04:34 +0000 Received: from DM6NAM11FT068.eop-nam11.prod.protection.outlook.com (2603:10b6:3:7b:cafe::5c) by DM5PR13CA0034.outlook.office365.com (2603:10b6:3:7b::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4373.7 via Frontend Transport; Thu, 22 Jul 2021 11:04:34 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.35) smtp.mailfrom=nvidia.com; kernel.org; dkim=none (message not signed) header.d=none; kernel.org; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.35 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.35; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.35) by DM6NAM11FT068.mail.protection.outlook.com (10.13.173.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4352.24 via Frontend Transport; Thu, 22 Jul 2021 11:04:33 +0000 Received: from HQMAIL105.nvidia.com (172.20.187.12) by HQMAIL111.nvidia.com (172.20.187.18) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Thu, 22 Jul 2021 11:04:33 +0000 Received: from HQMAIL107.nvidia.com (172.20.187.13) by HQMAIL105.nvidia.com (172.20.187.12) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Thu, 22 Jul 2021 11:04:33 +0000 Received: from vdi.nvidia.com (172.20.187.5) by mail.nvidia.com (172.20.187.13) with Microsoft SMTP Server id 15.0.1497.2 via Frontend Transport; Thu, 22 Jul 2021 11:04:29 +0000 From: Boris Pismenny To: , , , , , , , , , , CC: , , , , , , Boris Pismenny Subject: [PATCH v5 net-next 04/36] net/tls: expose get_netdev_for_sock Date: Thu, 22 Jul 2021 14:02:53 +0300 Message-ID: <20210722110325.371-5-borisp@nvidia.com> X-Mailer: git-send-email 2.24.1 In-Reply-To: <20210722110325.371-1-borisp@nvidia.com> References: <20210722110325.371-1-borisp@nvidia.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: ed58ca19-fe22-449b-c50e-08d94d007cd5 X-MS-TrafficTypeDiagnostic: BY5PR12MB4194: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:192; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: GnlUnBv+nZjtywZL3KVjeGe34GNEkd59T/6o7G+AZvwe9N5nXQTCDRPUWgcBQhYvLCIuUjqg55FOucQGHepAV6BH7ZfTVF6fmxl4NOyCzUb0HzY1u3NF/kLKI8P8Yuj5SKtu+oEXJ8zHZ8zHkp/230i2DatYZZ8BraTLAb3IUQypfukd/0gcuWZ01DUIZvDlBZNvEGdbeZYauBVSVX7x5TVq+yXX8RL0u7tm5JJAjRfUquWr4qwJ5SaavBWcCYFemDLRQOyqx4JBus0PHcYFXW8pvRaHvbc6LmPrB382gr5PuJh2BzRlbgn9Z7Lq/kjZ2xwXkz0aDadCZNlbTCKgeutKHAH2jDvgPuGq91P5Rw6W1tZMBoxtOXvM58a9gWLiVcrn5exvdDdHCtr2KOVfT36VfiBe+5dXY9ofTXKhispPcZ2fWoENgwE7TH9Llq0Dk+gMG5uA+gotWY6Yiv3o+IS6MQvF7ExTJVKevKZnbvW+sbZoYipfN3Tpv5H5AqY6aVjH+zWF11kPAwQ/A4DNpGJHc2ZoR+1rAkxBjADiYI4LfsRlI+v5D23daSorr25/ItTSd5jT5FUbIKghTzvxMPcmWJfqmDFxaTcolml/BVayHneOPpKIXAGTU1JTUJGDCLOC2bnRLDFxk5uJWzntoyXTfzuce8wwgNamo8I1jRzWUxqkradRXFVeT+N0Tx7pBUCMjkbR3p6QBGRDFmf/UWcitpFwfUjG4yp8xIdgs+w= X-Forefront-Antispam-Report: CIP:216.228.112.35; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:schybrid02.nvidia.com; CAT:NONE; SFS:(4636009)(136003)(346002)(39860400002)(396003)(376002)(46966006)(36840700001)(478600001)(8676002)(6666004)(36860700001)(83380400001)(356005)(47076005)(921005)(36906005)(316002)(70206006)(86362001)(110136005)(54906003)(5660300002)(7416002)(70586007)(336012)(4326008)(1076003)(2906002)(186003)(8936002)(2616005)(107886003)(26005)(7696005)(426003)(36756003)(82310400003)(82740400003)(7636003); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Jul 2021 11:04:33.8389 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: ed58ca19-fe22-449b-c50e-08d94d007cd5 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.112.35]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DM6NAM11FT068.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY5PR12MB4194 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Boris Pismenny get_netdev_for_sock is a utility that is used to obtain the net_device structure from a connected socket. Later patches will use this for nvme-tcp DDP and DDP DDGST offloads. Signed-off-by: Boris Pismenny Reviewed-by: Sagi Grimberg --- include/net/sock.h | 17 +++++++++++++++++ net/tls/tls_device.c | 20 ++------------------ 2 files changed, 19 insertions(+), 18 deletions(-) diff --git a/include/net/sock.h b/include/net/sock.h index 8bdd80027ffb..5fa2fd192d18 100644 --- a/include/net/sock.h +++ b/include/net/sock.h @@ -2769,4 +2769,21 @@ void sock_set_sndtimeo(struct sock *sk, s64 secs); int sock_bind_add(struct sock *sk, struct sockaddr *addr, int addr_len); +/* Assume that the socket is already connected */ +static inline struct net_device *get_netdev_for_sock(struct sock *sk, bool hold) +{ + struct dst_entry *dst = sk_dst_get(sk); + struct net_device *netdev = NULL; + + if (likely(dst)) { + netdev = dst->dev; + if (hold) + dev_hold(netdev); + } + + dst_release(dst); + + return netdev; +} + #endif /* _SOCK_H */ diff --git a/net/tls/tls_device.c b/net/tls/tls_device.c index b932469ee69c..06aa5f1e73d1 100644 --- a/net/tls/tls_device.c +++ b/net/tls/tls_device.c @@ -107,22 +107,6 @@ static void tls_device_queue_ctx_destruction(struct tls_context *ctx) spin_unlock_irqrestore(&tls_device_lock, flags); } -/* We assume that the socket is already connected */ -static struct net_device *get_netdev_for_sock(struct sock *sk) -{ - struct dst_entry *dst = sk_dst_get(sk); - struct net_device *netdev = NULL; - - if (likely(dst)) { - netdev = netdev_sk_get_lowest_dev(dst->dev, sk); - dev_hold(netdev); - } - - dst_release(dst); - - return netdev; -} - static void destroy_record(struct tls_record_info *record) { int i; @@ -1118,7 +1102,7 @@ int tls_set_device_offload(struct sock *sk, struct tls_context *ctx) if (skb) TCP_SKB_CB(skb)->eor = 1; - netdev = get_netdev_for_sock(sk); + netdev = get_netdev_for_sock(sk, true); if (!netdev) { pr_err_ratelimited("%s: netdev not found\n", __func__); rc = -EINVAL; @@ -1194,7 +1178,7 @@ int tls_set_device_offload_rx(struct sock *sk, struct tls_context *ctx) if (ctx->crypto_recv.info.version != TLS_1_2_VERSION) return -EOPNOTSUPP; - netdev = get_netdev_for_sock(sk); + netdev = get_netdev_for_sock(sk, true); if (!netdev) { pr_err_ratelimited("%s: netdev not found\n", __func__); return -EINVAL; From patchwork Thu Jul 22 11:02:54 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Boris Pismenny X-Patchwork-Id: 485110 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-20.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER, INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 57CA3C63793 for ; Thu, 22 Jul 2021 11:04:58 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 4045260725 for ; Thu, 22 Jul 2021 11:04:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231708AbhGVKYV (ORCPT ); Thu, 22 Jul 2021 06:24:21 -0400 Received: from mail-bn8nam11on2042.outbound.protection.outlook.com ([40.107.236.42]:29369 "EHLO NAM11-BN8-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S231694AbhGVKYF (ORCPT ); Thu, 22 Jul 2021 06:24:05 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=nNuy9ZxXoc11r1+N6QV4ieKvp31jjg3bPu8pCnsXbzYFLKKvimU8MDxs/M2rwbTQoKeoTm3HB1mZ21BvTai1YzRZXpfoLaaGi62IYQ31gVWVRTAfE0QqpqIyPiRokeWYwmIvoG5fERixVyG4juN9fqp7ovduZM1dXIdtN+1aLSoA60a2ukR2ksFbpBRmIzfbAFcGYvXmyd0bu8Ymel3TvN1rRfw+pO+03zK0H4xqgy5FO9qeHO+qZUvvVy3604f2OT6dht010T3gSslO/PFjiJhIqaI19Qp83PUXAdTAB4rE6O0P0iu1VBR71okN6YVnUNTq8Vumg7aEIZOvytDvdQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=MtBrg4VgmD3102l6GupTqKopXJPsV6z+Jgoz23AiywA=; b=L4c5+LlZhaSGzW/lveA35saJc6wzFbJGLsmZgwaALYNU0aswBwRKgYwCU5+4EDN9t1+3VPiR2TYk0/Eb70/v+QUMF+KAEh+CUoZYIrzVURYL3eDpmvHrXpLj0sLsd4NfNYzh5/5RXyx9tBLQSyspM/PtTzITlXicQy2bFgKPmSrxrM6By2ie442L23fPVi4E6DD5BdwSBzul9AkTi1JZ/g18nVGWXpyrQs87qQEmlgVm/PP83TYLAXkuUmMijaIR7aasg18l1K5ETO+fvIeMqVeGYRKCZw63i/SQM7HS6HUAmUyM5O3qRZPPCM3NOUz+IREjg7cEfBImZIrPaMWG2A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.36) smtp.rcpttodomain=google.com smtp.mailfrom=nvidia.com; dmarc=pass (p=quarantine sp=none pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=MtBrg4VgmD3102l6GupTqKopXJPsV6z+Jgoz23AiywA=; b=I/t9vpyErC16bPVkpz0cyo1niN2M9eVefC7MXfanbIH+gvna5PNJhS7fBnjCe8rNHoYq3gcW1tjAoxMH53u3VntOKVuAI0k3ntWGFJBKBmUVkfi8miTdafY9G4pDhl+8m07avYU7M9AWXQ+ST9fHiQqlh8xWzgddSdx39+YbZAhCZitgQM7jyNKNHX+/uqY+d1tXwvBgx3V1uTASwr+8KMYutShJ2hP6k1GRPWOk21x84DELPxVizaKCAzfOzMZab8pOClf9pc2SScstqtmbWrbgq/n6Jt52fPXwKgAJw0HsTd+sl12+EvIJAAh2SYCQ8588MWIkU2Z5MH/RDZluvw== Received: from BN0PR02CA0053.namprd02.prod.outlook.com (2603:10b6:408:e5::28) by SN1PR12MB2557.namprd12.prod.outlook.com (2603:10b6:802:22::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4352.25; Thu, 22 Jul 2021 11:04:39 +0000 Received: from BN8NAM11FT055.eop-nam11.prod.protection.outlook.com (2603:10b6:408:e5:cafe::46) by BN0PR02CA0053.outlook.office365.com (2603:10b6:408:e5::28) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4352.25 via Frontend Transport; Thu, 22 Jul 2021 11:04:39 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.36) smtp.mailfrom=nvidia.com; google.com; dkim=none (message not signed) header.d=none; google.com; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.36 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.36; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.36) by BN8NAM11FT055.mail.protection.outlook.com (10.13.177.62) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4352.24 via Frontend Transport; Thu, 22 Jul 2021 11:04:38 +0000 Received: from HQMAIL105.nvidia.com (172.20.187.12) by HQMAIL101.nvidia.com (172.20.187.10) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Thu, 22 Jul 2021 11:04:38 +0000 Received: from HQMAIL107.nvidia.com (172.20.187.13) by HQMAIL105.nvidia.com (172.20.187.12) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Thu, 22 Jul 2021 11:04:38 +0000 Received: from vdi.nvidia.com (172.20.187.5) by mail.nvidia.com (172.20.187.13) with Microsoft SMTP Server id 15.0.1497.2 via Frontend Transport; Thu, 22 Jul 2021 11:04:33 +0000 From: Boris Pismenny To: , , , , , , , , , , CC: , , , , , , Boris Pismenny , Ben Ben-Ishay , Or Gerlitz , Yoray Zack Subject: [PATCH v5 net-next 05/36] nvme-tcp: Add DDP offload control path Date: Thu, 22 Jul 2021 14:02:54 +0300 Message-ID: <20210722110325.371-6-borisp@nvidia.com> X-Mailer: git-send-email 2.24.1 In-Reply-To: <20210722110325.371-1-borisp@nvidia.com> References: <20210722110325.371-1-borisp@nvidia.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 06f67630-6359-41e3-6df9-08d94d007ff0 X-MS-TrafficTypeDiagnostic: SN1PR12MB2557: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:8; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: awCoTXfpfwOjGNc8luNrUwePr8R1o9/QhKK+n0GGtwg0ZKU+uTcjkIl0FEDoLr2zZA2Ni6ebaR480uB0pxZfzFqwvA/7msh+l7TGM+g8Sp8Cpi4PgJfRbCfUBqkVEavcYw8qdCfExMQZjTaiB0fgcwkVxr+Zz+D96sXF2luvYBEQCR3Xew4Riut0D4RY6HtXe2cE8p4gyOZafIb9HO2imUhA5MfpO2WxTc3nVB1J1nY0qA0A8WTnNow4+AQ8+Ld29xf0UhyBu3PnXrZ1B7JN30axAJZJDCdTHHHcWxYT+mWuikWWfbYXFrUrO7VcSULmLI3FH/jPeGvM4ooitI4TwIPJff7dp1IoZqTZBrqpM40sDfid9n44t1ScYzMad59+NeSNZapZpz/NNiZ8ToY8rv0yEcJR33Ds2HGO9cyfvyZepuikxuwfoIX7FhuGV6SeJ8UTvlT6ZbvYaFK6ZgE5GQAw3Y+eFdOLxzGqyl3nb+/qTI1KypJHt69nJ5/3oaLUQDwPVzFyWPH7xJ4curlGVFr7k+e09HrjpqzK6RkIBgf8WSsVEH2O4BdxE2NkdLSZjklFm3458WCzlLxidXGvdGHMRil/bEPauYA4d6djpxL8lf6PpQjYESzE5Sq6iK0/K21FMfsNSpzI6bj4Y2EjnVI1Q37uocnGczxoKEoiIMG1bp/UXfgjJxTb2Qt2TW3CykhAGIdkZxKzip0n+Tv01Q08B1ACHVjVSpcVp0w4YNs= X-Forefront-Antispam-Report: CIP:216.228.112.36; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:schybrid05.nvidia.com; CAT:NONE; SFS:(4636009)(396003)(376002)(39860400002)(136003)(346002)(36840700001)(46966006)(86362001)(36860700001)(2906002)(70586007)(47076005)(2616005)(110136005)(36906005)(4326008)(36756003)(54906003)(8676002)(70206006)(7696005)(478600001)(336012)(7636003)(186003)(316002)(356005)(107886003)(1076003)(5660300002)(82740400003)(83380400001)(921005)(30864003)(7416002)(26005)(8936002)(6666004)(82310400003)(426003); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Jul 2021 11:04:38.9269 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 06f67630-6359-41e3-6df9-08d94d007ff0 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.112.36]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BN8NAM11FT055.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN1PR12MB2557 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Boris Pismenny This commit introduces direct data placement offload to NVME TCP. There is a context per queue, which is established after the handshake using the ulp_ddp_sk_add/del NDOs. Additionally, a resynchronization routine is used to assist hardware recovery from TCP OOO, and continue the offload. Resynchronization operates as follows: 1. TCP OOO causes the NIC HW to stop the offload 2. NIC HW identifies a PDU header at some TCP sequence number, and asks NVMe-TCP to confirm it. This request is delivered from the NIC driver to NVMe-TCP by first finding the socket for the packet that triggered the request, and then finding the nvme_tcp_queue that is used by this routine. Finally, the request is recorded in the nvme_tcp_queue. 3. When NVMe-TCP observes the requested TCP sequence, it will compare it with the PDU header TCP sequence, and report the result to the NIC driver (ulp_ddp_resync), which will update the HW, and resume offload when all is successful. Furthermore, we let the offloading driver advertise what is the max hw sectors/segments via ulp_ddp_limits. A follow-up patch introduces the data-path changes required for this offload. Signed-off-by: Boris Pismenny Signed-off-by: Ben Ben-Ishay Signed-off-by: Or Gerlitz Signed-off-by: Yoray Zack --- drivers/nvme/host/tcp.c | 180 +++++++++++++++++++++++++++++++++++++++- include/linux/skbuff.h | 4 +- net/core/datagram.c | 4 +- 3 files changed, 182 insertions(+), 6 deletions(-) diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c index c7bd37103cf4..f1a5520cabec 100644 --- a/drivers/nvme/host/tcp.c +++ b/drivers/nvme/host/tcp.c @@ -14,6 +14,7 @@ #include #include #include +#include #include "nvme.h" #include "fabrics.h" @@ -62,6 +63,7 @@ enum nvme_tcp_queue_flags { NVME_TCP_Q_ALLOCATED = 0, NVME_TCP_Q_LIVE = 1, NVME_TCP_Q_POLLING = 2, + NVME_TCP_Q_OFF_DDP = 3, }; enum nvme_tcp_recv_state { @@ -111,6 +113,8 @@ struct nvme_tcp_queue { void (*state_change)(struct sock *); void (*data_ready)(struct sock *); void (*write_space)(struct sock *); + + atomic64_t resync_req; }; struct nvme_tcp_ctrl { @@ -130,6 +134,8 @@ struct nvme_tcp_ctrl { struct delayed_work connect_work; struct nvme_tcp_request async_req; u32 io_queues[HCTX_MAX_TYPES]; + + struct net_device *offloading_netdev; }; static LIST_HEAD(nvme_tcp_ctrl_list); @@ -219,6 +225,167 @@ static inline size_t nvme_tcp_pdu_last_send(struct nvme_tcp_request *req, return nvme_tcp_pdu_data_left(req) <= len; } +#ifdef CONFIG_ULP_DDP + +static bool nvme_tcp_resync_request(struct sock *sk, u32 seq, u32 flags); +static const struct ulp_ddp_ulp_ops nvme_tcp_ddp_ulp_ops = { + .resync_request = nvme_tcp_resync_request, +}; + +static int nvme_tcp_offload_socket(struct nvme_tcp_queue *queue) +{ + struct net_device *netdev = queue->ctrl->offloading_netdev; + struct nvme_tcp_ddp_config config = {}; + int ret; + + if (!netdev || !(netdev->features & NETIF_F_HW_ULP_DDP)) + return -EOPNOTSUPP; + + config.cfg.type = ULP_DDP_NVME; + config.pfv = NVME_TCP_PFV_1_0; + config.cpda = 0; + config.dgst = queue->hdr_digest ? + NVME_TCP_HDR_DIGEST_ENABLE : 0; + config.dgst |= queue->data_digest ? + NVME_TCP_DATA_DIGEST_ENABLE : 0; + config.queue_size = queue->queue_size; + config.queue_id = nvme_tcp_queue_id(queue); + config.io_cpu = queue->io_cpu; + + dev_hold(netdev); /* put by unoffload_socket */ + ret = netdev->ulp_ddp_ops->ulp_ddp_sk_add(netdev, + queue->sock->sk, + &config.cfg); + if (ret) { + dev_put(netdev); + return ret; + } + + inet_csk(queue->sock->sk)->icsk_ulp_ddp_ops = &nvme_tcp_ddp_ulp_ops; + if (netdev->features & NETIF_F_HW_ULP_DDP) + set_bit(NVME_TCP_Q_OFF_DDP, &queue->flags); + + return ret; +} + +static void nvme_tcp_unoffload_socket(struct nvme_tcp_queue *queue) +{ + struct net_device *netdev = queue->ctrl->offloading_netdev; + + if (!netdev) { + dev_info_ratelimited(queue->ctrl->ctrl.device, "netdev not found\n"); + return; + } + + clear_bit(NVME_TCP_Q_OFF_DDP, &queue->flags); + + netdev->ulp_ddp_ops->ulp_ddp_sk_del(netdev, queue->sock->sk); + + inet_csk(queue->sock->sk)->icsk_ulp_ddp_ops = NULL; + dev_put(netdev); /* held by offload_socket */ +} + +static int nvme_tcp_offload_limits(struct nvme_tcp_queue *queue) +{ + struct net_device *netdev = get_netdev_for_sock(queue->sock->sk, true); + struct ulp_ddp_limits limits; + int ret = 0; + + if (!netdev) { + dev_info_ratelimited(queue->ctrl->ctrl.device, "netdev not found\n"); + queue->ctrl->offloading_netdev = NULL; + return -ENODEV; + } + + if ((netdev->features & NETIF_F_HW_ULP_DDP) && + netdev->ulp_ddp_ops && + netdev->ulp_ddp_ops->ulp_ddp_limits) + ret = netdev->ulp_ddp_ops->ulp_ddp_limits(netdev, &limits); + else + ret = -EOPNOTSUPP; + + if (!ret) { + queue->ctrl->offloading_netdev = netdev; + dev_dbg_ratelimited(queue->ctrl->ctrl.device, + "netdev %s offload limits: max_ddp_sgl_len %d\n", + netdev->name, limits.max_ddp_sgl_len); + queue->ctrl->ctrl.max_segments = limits.max_ddp_sgl_len; + queue->ctrl->ctrl.max_hw_sectors = + limits.max_ddp_sgl_len << (ilog2(SZ_4K) - 9); + } else { + queue->ctrl->offloading_netdev = NULL; + } + + /* release the device as no offload context is established yet. */ + dev_put(netdev); + + return ret; +} + +static void nvme_tcp_resync_response(struct nvme_tcp_queue *queue, + struct sk_buff *skb, unsigned int offset) +{ + u64 pdu_seq = TCP_SKB_CB(skb)->seq + offset - queue->pdu_offset; + struct net_device *netdev = queue->ctrl->offloading_netdev; + u64 pdu_val = (pdu_seq << 32) | ULP_DDP_RESYNC_REQ; + u64 resync_val; + u32 resync_seq; + + resync_val = atomic64_read(&queue->resync_req); + /* Lower 32 bit flags. Check validity of the request */ + if ((resync_val & ULP_DDP_RESYNC_REQ) == 0) + return; + + /* Obtain and check requested sequence number: is this PDU header before the request? */ + resync_seq = resync_val >> 32; + if (before(pdu_seq, resync_seq)) + return; + + if (unlikely(!netdev)) { + pr_info_ratelimited("%s: netdev not found\n", __func__); + return; + } + + /** + * The atomic operation gurarantees that we don't miss any NIC driver + * resync requests submitted after the above checks. + */ + if (atomic64_cmpxchg(&queue->resync_req, pdu_val, + pdu_val & ~ULP_DDP_RESYNC_REQ) != atomic64_read(&queue->resync_req)) + netdev->ulp_ddp_ops->ulp_ddp_resync(netdev, queue->sock->sk, pdu_seq); +} + +static bool nvme_tcp_resync_request(struct sock *sk, u32 seq, u32 flags) +{ + struct nvme_tcp_queue *queue = sk->sk_user_data; + + atomic64_set(&queue->resync_req, + (((uint64_t)seq << 32) | flags)); + + return true; +} + +#else + +static int nvme_tcp_offload_socket(struct nvme_tcp_queue *queue) +{ + return -EINVAL; +} + +static void nvme_tcp_unoffload_socket(struct nvme_tcp_queue *queue) +{} + +static int nvme_tcp_offload_limits(struct nvme_tcp_queue *queue) +{ + return -EINVAL; +} + +static void nvme_tcp_resync_response(struct nvme_tcp_queue *queue, + struct sk_buff *skb, unsigned int offset) +{} + +#endif + static void nvme_tcp_init_iter(struct nvme_tcp_request *req, unsigned int dir) { @@ -649,6 +816,9 @@ static int nvme_tcp_recv_pdu(struct nvme_tcp_queue *queue, struct sk_buff *skb, size_t rcv_len = min_t(size_t, *len, queue->pdu_remaining); int ret; + if (test_bit(NVME_TCP_Q_OFF_DDP, &queue->flags)) + nvme_tcp_resync_response(queue, skb, *offset); + ret = skb_copy_bits(skb, *offset, &pdu[queue->pdu_offset], rcv_len); if (unlikely(ret)) @@ -1555,6 +1725,9 @@ static void __nvme_tcp_stop_queue(struct nvme_tcp_queue *queue) kernel_sock_shutdown(queue->sock, SHUT_RDWR); nvme_tcp_restore_sock_calls(queue); cancel_work_sync(&queue->io_work); + + if (test_bit(NVME_TCP_Q_OFF_DDP, &queue->flags)) + nvme_tcp_unoffload_socket(queue); } static void nvme_tcp_stop_queue(struct nvme_ctrl *nctrl, int qid) @@ -1573,10 +1746,13 @@ static int nvme_tcp_start_queue(struct nvme_ctrl *nctrl, int idx) struct nvme_tcp_ctrl *ctrl = to_tcp_ctrl(nctrl); int ret; - if (idx) + if (idx) { ret = nvmf_connect_io_queue(nctrl, idx, false); - else + nvme_tcp_offload_socket(&ctrl->queues[idx]); + } else { ret = nvmf_connect_admin_queue(nctrl); + nvme_tcp_offload_limits(&ctrl->queues[idx]); + } if (!ret) { set_bit(NVME_TCP_Q_LIVE, &ctrl->queues[idx].flags); diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h index 8c1bfd7081d1..55dc858ff349 100644 --- a/include/linux/skbuff.h +++ b/include/linux/skbuff.h @@ -3613,7 +3613,7 @@ __poll_t datagram_poll(struct file *file, struct socket *sock, struct poll_table_struct *wait); int skb_copy_datagram_iter(const struct sk_buff *from, int offset, struct iov_iter *to, int size); -#ifdef CONFIG_TCP_DDP +#ifdef CONFIG_ULP_DDP int skb_ddp_copy_datagram_iter(const struct sk_buff *from, int offset, struct iov_iter *to, int size); #endif @@ -3627,7 +3627,7 @@ int skb_copy_and_csum_datagram_msg(struct sk_buff *skb, int hlen, int skb_copy_and_hash_datagram_iter(const struct sk_buff *skb, int offset, struct iov_iter *to, int len, struct ahash_request *hash); -#ifdef CONFIG_TCP_DDP +#ifdef CONFIG_ULP_DDP int skb_ddp_copy_and_hash_datagram_iter(const struct sk_buff *skb, int offset, struct iov_iter *to, int len, struct ahash_request *hash); diff --git a/net/core/datagram.c b/net/core/datagram.c index d346fd5da22c..5ad5fb22d3f8 100644 --- a/net/core/datagram.c +++ b/net/core/datagram.c @@ -495,7 +495,7 @@ static int __skb_datagram_iter(const struct sk_buff *skb, int offset, return 0; } -#ifdef CONFIG_TCP_DDP +#ifdef CONFIG_ULP_DDP /** * skb_ddp_copy_and_hash_datagram_iter - Copies datagrams from skb frags to * an iterator and update a hash. If the iterator and skb frag point to the @@ -534,7 +534,7 @@ int skb_copy_and_hash_datagram_iter(const struct sk_buff *skb, int offset, } EXPORT_SYMBOL(skb_copy_and_hash_datagram_iter); -#ifdef CONFIG_TCP_DDP +#ifdef CONFIG_ULP_DDP static size_t simple_ddp_copy_to_iter(const void *addr, size_t bytes, void *data __always_unused, struct iov_iter *i) From patchwork Thu Jul 22 11:02:55 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Boris Pismenny X-Patchwork-Id: 484323 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-20.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER, INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B2FDEC6377D for ; Thu, 22 Jul 2021 11:05:06 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 908596120D for ; Thu, 22 Jul 2021 11:05:06 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231722AbhGVKYZ (ORCPT ); Thu, 22 Jul 2021 06:24:25 -0400 Received: from mail-mw2nam12on2043.outbound.protection.outlook.com ([40.107.244.43]:51168 "EHLO NAM12-MW2-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S231706AbhGVKYL (ORCPT ); Thu, 22 Jul 2021 06:24:11 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=e7L7ajaAez1yLPkcuLhP5ZRgiliRQ5B70vKS7shDFdEQo7pmJM8vniROnsy+AfKoLiHaSjDCyCTsbklCY972aYSs1+IruuS3mXnt7vW4+rE+mZQ6NEus1R20Xi+HBeA09IaDUaantiMYsXO6hl4EuQ4sDwEVTDa24S1/iLhlPfwIRUaLKVPShCn+nVU7b0f2I856n5rDyAvRjQYxCCvjj5SOMtfTC74Ht23UrEQJR3N8N+gc/C9JdZYoat7YjQz7dxTk10rIXULZ59t6YcilNSYBIGlATu0wKDrhZaRWDZUDBp6OCLm4HXXBmsJl9cKjztx3sQFPZmqO7JQIdvrPRw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=6ivLTFyfz8S9XoKV9ew/wjIE3HWri6kJ+VYDIRqosno=; b=KIjYOqgiLTwNMjzMlyRxyCtwAsrrKcSa+HEbrMM24bYRPVlVcYmaAtDevfnx7LFwq/GzFiWnb9wTBtKHpA4UsK3QVRJpJ7G2S5yGhl+18Srn+sVDLsAzbm7tELmcK509w9OfvZd1yizT7YuQCr9vnpaeFHZABAVDWoBq3g4twjiA6C9QV9xYljSuFGqjZCfmxyGGZRnSQXw/vDJIP+GUTxvyQYt5LklBs0Xhqzc8snH4gZOwKELD+vq+c5TYxO4bhMGpf0ZdUEf9++44s3WGGOhqGwxaJuBbfQTGqHu9BcEopdcJ8bJefEmcnGblV1Vq2APkKN3taTWPh6XjzowDUg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.34) smtp.rcpttodomain=marvell.com smtp.mailfrom=nvidia.com; dmarc=pass (p=quarantine sp=none pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=6ivLTFyfz8S9XoKV9ew/wjIE3HWri6kJ+VYDIRqosno=; b=pTvYLTIahaKsOl7go1dl6uGCCCyiqH32MQ7Gobs+uuwd6t9ymJ2oxnbSfJX8PebQnIbYdWZniWFxGMAhw7RMhyHTNmuvxJnR1iPiCUS3ROwavUyB+NFq+yl45ymrYqWRgdEupeU7Iub7cYeYzgcHuFxsWzSGZ+B33Ws0ZczNWz0mvvGeRsdye7BiEPUmhqqCMXTpnnl3WJr5hSzPBPpZ9l2YGJt8A2eafYjVVw/DqoSGOAhlb6K4F5gu8+iM4IJZhu4I4rEOOawRtnhs/DHSXA7i8I7BcofuETW6f1TvrpvPfho5QozUpYQSLqfPMW3wmtsHnlb8ixo9qQfvvhgR5w== Received: from BN6PR21CA0002.namprd21.prod.outlook.com (2603:10b6:404:8e::12) by DM5PR1201MB0121.namprd12.prod.outlook.com (2603:10b6:4:56::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4331.29; Thu, 22 Jul 2021 11:04:43 +0000 Received: from BN8NAM11FT017.eop-nam11.prod.protection.outlook.com (2603:10b6:404:8e:cafe::79) by BN6PR21CA0002.outlook.office365.com (2603:10b6:404:8e::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4373.2 via Frontend Transport; Thu, 22 Jul 2021 11:04:43 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.34) smtp.mailfrom=nvidia.com; marvell.com; dkim=none (message not signed) header.d=none; marvell.com; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.34 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.34; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.34) by BN8NAM11FT017.mail.protection.outlook.com (10.13.177.93) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4352.24 via Frontend Transport; Thu, 22 Jul 2021 11:04:43 +0000 Received: from HQMAIL107.nvidia.com (172.20.187.13) by HQMAIL107.nvidia.com (172.20.187.13) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Thu, 22 Jul 2021 11:04:43 +0000 Received: from vdi.nvidia.com (172.20.187.5) by mail.nvidia.com (172.20.187.13) with Microsoft SMTP Server id 15.0.1497.2 via Frontend Transport; Thu, 22 Jul 2021 11:04:38 +0000 From: Boris Pismenny To: , , , , , , , , , , CC: , , , , , , Boris Pismenny , Ben Ben-Ishay , Or Gerlitz , Yoray Zack Subject: [PATCH v5 net-next 06/36] nvme-tcp: Add DDP data-path Date: Thu, 22 Jul 2021 14:02:55 +0300 Message-ID: <20210722110325.371-7-borisp@nvidia.com> X-Mailer: git-send-email 2.24.1 In-Reply-To: <20210722110325.371-1-borisp@nvidia.com> References: <20210722110325.371-1-borisp@nvidia.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 109c339d-6529-4726-d122-08d94d0082a0 X-MS-TrafficTypeDiagnostic: DM5PR1201MB0121: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:4303; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: NGTjr/F5feO6RdBg6BvxQXmXvHNQr9rH4Ac1ODSaJ7gNXkrDfI8gqqK4RTu9eJ5D0JGkkvO19R5yVg4hfhH+OiSMV3fe4AO4CzX3ncYuZQSGlT4dM7Of9dqzvaoIdgx4nG4u8/Hj/kN7guwBgJTaz8WCIVvHjuV9WjxjcSzxAZ3P5e+w98Hjy16fvWR99/T3RkvFy/FPqFk0LfQif7tIagcoPION5K7YImH6TYkXXZ+oRSRSoiPjI8ToWQ+ghMIiuVqbFDm1M3+fh62DewC84jSiCPi/UeGjAfrlhUugb/wb8jCV+RylHPHaa72AUYf5owjZXEVDlfOJXzJqt+6rJbr6ZLc+eB6yOZ1hzjuD1qcmPxiIHb/6P8MH07C1sR7/e1g50mAunEwwalvw1QAd4WGJ8Z8HtQhD7ZFdDYrYrSIbo3FiUpQ87gpTUTstPFrFbclzT8p7D40MLMEpgCxDD37HffCon3wvTKVaoSmNx6Rn5/GmTCQiFuHkOYsw2s1KxiOiIf6SpQoHXFL/s3Vu6fmPY6PwdxNZnTa0PZnJp0Em6VTsQqOnhSmopypQ1aDVH4ZRH7cfoJwcsCX2qF7UVtcJzTtk5zdCKfkqHH8Ul/j4uMyS/OjfgFiLuOXmCCwyA4HnQDmR4MlY9Odrgj8MbsTh2r2/Rt7DLOM+g6hdmEXcz64PNoHLSJsUnqEHqz7TtOHEw/h19jjVhxgsqGjDc0LClhVm1yJc44CwocXHhhk= X-Forefront-Antispam-Report: CIP:216.228.112.34; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:schybrid03.nvidia.com; CAT:NONE; SFS:(4636009)(39860400002)(136003)(376002)(396003)(346002)(46966006)(36840700001)(186003)(7416002)(8936002)(36906005)(336012)(70206006)(478600001)(70586007)(2616005)(426003)(6666004)(82310400003)(110136005)(54906003)(5660300002)(316002)(83380400001)(36860700001)(7696005)(107886003)(86362001)(2906002)(921005)(1076003)(4326008)(7636003)(36756003)(8676002)(26005)(356005)(47076005)(82740400003); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Jul 2021 11:04:43.4972 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 109c339d-6529-4726-d122-08d94d0082a0 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.112.34]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BN8NAM11FT017.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM5PR1201MB0121 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Boris Pismenny Introduce the NVMe-TCP DDP data-path offload. Using this interface, the NIC hardware will scatter TCP payload directly to the BIO pages according to the command_id in the PDU. To maintain the correctness of the network stack, the driver is expected to construct SKBs that point to the BIO pages. The data-path interface contains two routines: tcp_ddp_setup/teardown. The setup provides the mapping from command_id to the request buffers, while the teardown removes this mapping. For efficiency, we introduce an asynchronous nvme completion, which is split between NVMe-TCP and the NIC driver as follows: NVMe-TCP performs the specific completion, while NIC driver performs the generic mq_blk completion. Signed-off-by: Boris Pismenny Signed-off-by: Ben Ben-Ishay Signed-off-by: Or Gerlitz Signed-off-by: Yoray Zack --- drivers/nvme/host/tcp.c | 150 ++++++++++++++++++++++++++++++++++++---- 1 file changed, 138 insertions(+), 12 deletions(-) diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c index f1a5520cabec..34982fb0c655 100644 --- a/drivers/nvme/host/tcp.c +++ b/drivers/nvme/host/tcp.c @@ -57,6 +57,11 @@ struct nvme_tcp_request { size_t offset; size_t data_sent; enum nvme_tcp_send_state state; + + bool offloaded; + struct ulp_ddp_io ddp; + __le16 status; + union nvme_result result; }; enum nvme_tcp_queue_flags { @@ -225,13 +230,76 @@ static inline size_t nvme_tcp_pdu_last_send(struct nvme_tcp_request *req, return nvme_tcp_pdu_data_left(req) <= len; } +static int nvme_tcp_req_map_sg(struct nvme_tcp_request *req, struct request *rq) +{ + int ret; + + req->ddp.sg_table.sgl = req->ddp.first_sgl; + ret = sg_alloc_table_chained(&req->ddp.sg_table, blk_rq_nr_phys_segments(rq), + req->ddp.sg_table.sgl, SG_CHUNK_SIZE); + if (ret) + return -ENOMEM; + req->ddp.nents = blk_rq_map_sg(rq->q, rq, req->ddp.sg_table.sgl); + return 0; +} + #ifdef CONFIG_ULP_DDP static bool nvme_tcp_resync_request(struct sock *sk, u32 seq, u32 flags); +static void nvme_tcp_ddp_teardown_done(void *ddp_ctx); static const struct ulp_ddp_ulp_ops nvme_tcp_ddp_ulp_ops = { .resync_request = nvme_tcp_resync_request, + .ddp_teardown_done = nvme_tcp_ddp_teardown_done, }; +static int nvme_tcp_teardown_ddp(struct nvme_tcp_queue *queue, + u16 command_id, + struct request *rq) +{ + struct nvme_tcp_request *req = blk_mq_rq_to_pdu(rq); + struct net_device *netdev = queue->ctrl->offloading_netdev; + int ret; + + ret = netdev->ulp_ddp_ops->ulp_ddp_teardown(netdev, queue->sock->sk, + &req->ddp, rq); + sg_free_table_chained(&req->ddp.sg_table, SG_CHUNK_SIZE); + return ret; +} + +static void nvme_tcp_ddp_teardown_done(void *ddp_ctx) +{ + struct request *rq = ddp_ctx; + struct nvme_tcp_request *req = blk_mq_rq_to_pdu(rq); + + if (!nvme_try_complete_req(rq, req->status, req->result)) + nvme_complete_rq(rq); +} + +static int nvme_tcp_setup_ddp(struct nvme_tcp_queue *queue, + u16 command_id, + struct request *rq) +{ + struct nvme_tcp_request *req = blk_mq_rq_to_pdu(rq); + struct net_device *netdev = queue->ctrl->offloading_netdev; + int ret; + + if (!test_bit(NVME_TCP_Q_OFF_DDP, &queue->flags) || + !blk_rq_nr_phys_segments(rq) || !(rq_data_dir(rq) == READ)) + return -EINVAL; + + req->ddp.command_id = command_id; + ret = nvme_tcp_req_map_sg(req, rq); + if (ret) + return -ENOMEM; + + ret = netdev->ulp_ddp_ops->ulp_ddp_setup(netdev, + queue->sock->sk, + &req->ddp); + if (!ret) + req->offloaded = true; + return ret; +} + static int nvme_tcp_offload_socket(struct nvme_tcp_queue *queue) { struct net_device *netdev = queue->ctrl->offloading_netdev; @@ -342,7 +410,7 @@ static void nvme_tcp_resync_response(struct nvme_tcp_queue *queue, return; if (unlikely(!netdev)) { - pr_info_ratelimited("%s: netdev not found\n", __func__); + dev_info_ratelimited(queue->ctrl->ctrl.device, "netdev not found\n"); return; } @@ -367,6 +435,20 @@ static bool nvme_tcp_resync_request(struct sock *sk, u32 seq, u32 flags) #else +static int nvme_tcp_setup_ddp(struct nvme_tcp_queue *queue, + u16 command_id, + struct request *rq) +{ + return -EINVAL; +} + +static int nvme_tcp_teardown_ddp(struct nvme_tcp_queue *queue, + u16 command_id, + struct request *rq) +{ + return -EINVAL; +} + static int nvme_tcp_offload_socket(struct nvme_tcp_queue *queue) { return -EINVAL; @@ -650,6 +732,24 @@ static void nvme_tcp_error_recovery(struct nvme_ctrl *ctrl) queue_work(nvme_reset_wq, &to_tcp_ctrl(ctrl)->err_work); } +static void nvme_tcp_complete_request(struct request *rq, + __le16 status, + union nvme_result result, + __u16 command_id) +{ + struct nvme_tcp_request *req = blk_mq_rq_to_pdu(rq); + struct nvme_tcp_queue *queue = req->queue; + + if (req->offloaded) { + req->status = status; + req->result = result; + nvme_tcp_teardown_ddp(queue, command_id, rq); + } else { + if (!nvme_try_complete_req(rq, status, result)) + nvme_complete_rq(rq); + } +} + static int nvme_tcp_process_nvme_cqe(struct nvme_tcp_queue *queue, struct nvme_completion *cqe) { @@ -664,10 +764,8 @@ static int nvme_tcp_process_nvme_cqe(struct nvme_tcp_queue *queue, return -EINVAL; } - if (!nvme_try_complete_req(rq, cqe->status, cqe->result)) - nvme_complete_rq(rq); + nvme_tcp_complete_request(rq, cqe->status, cqe->result, cqe->command_id); queue->nr_cqe++; - return 0; } @@ -863,10 +961,39 @@ static int nvme_tcp_recv_pdu(struct nvme_tcp_queue *queue, struct sk_buff *skb, static inline void nvme_tcp_end_request(struct request *rq, u16 status) { + struct nvme_tcp_request *req = blk_mq_rq_to_pdu(rq); + struct nvme_tcp_queue *queue = req->queue; + struct nvme_tcp_data_pdu *pdu = (void *)queue->pdu; union nvme_result res = {}; - if (!nvme_try_complete_req(rq, cpu_to_le16(status << 1), res)) - nvme_complete_rq(rq); + nvme_tcp_complete_request(rq, cpu_to_le16(status << 1), res, pdu->command_id); +} + + +static int nvme_tcp_consume_skb(struct nvme_tcp_queue *queue, struct sk_buff *skb, + unsigned int *offset, struct iov_iter *iter, int recv_len) +{ + int ret; + +#ifdef CONFIG_ULP_DDP + if (test_bit(NVME_TCP_Q_OFF_DDP, &queue->flags)) { + if (queue->data_digest) + ret = skb_ddp_copy_and_hash_datagram_iter(skb, *offset, iter, recv_len, + queue->rcv_hash); + else + ret = skb_ddp_copy_datagram_iter(skb, *offset, iter, recv_len); + } else { +#endif + if (queue->data_digest) + ret = skb_copy_and_hash_datagram_iter(skb, *offset, iter, recv_len, + queue->rcv_hash); + else + ret = skb_copy_datagram_iter(skb, *offset, iter, recv_len); +#ifdef CONFIG_ULP_DDP + } +#endif + + return ret; } static int nvme_tcp_recv_data(struct nvme_tcp_queue *queue, struct sk_buff *skb, @@ -913,12 +1040,7 @@ static int nvme_tcp_recv_data(struct nvme_tcp_queue *queue, struct sk_buff *skb, recv_len = min_t(size_t, recv_len, iov_iter_count(&req->iter)); - if (queue->data_digest) - ret = skb_copy_and_hash_datagram_iter(skb, *offset, - &req->iter, recv_len, queue->rcv_hash); - else - ret = skb_copy_datagram_iter(skb, *offset, - &req->iter, recv_len); + ret = nvme_tcp_consume_skb(queue, skb, offset, &req->iter, recv_len); if (ret) { dev_err(queue->ctrl->ctrl.device, "queue %d failed to copy request %#x data", @@ -1142,6 +1264,7 @@ static int nvme_tcp_try_send_cmd_pdu(struct nvme_tcp_request *req) bool inline_data = nvme_tcp_has_inline_data(req); u8 hdgst = nvme_tcp_hdgst_len(queue); int len = sizeof(*pdu) + hdgst - req->offset; + struct request *rq = blk_mq_rq_from_pdu(req); int flags = MSG_DONTWAIT; int ret; @@ -1150,6 +1273,8 @@ static int nvme_tcp_try_send_cmd_pdu(struct nvme_tcp_request *req) else flags |= MSG_EOR; + nvme_tcp_setup_ddp(queue, pdu->cmd.common.command_id, rq); + if (queue->hdr_digest && !req->offset) nvme_tcp_hdgst(queue->snd_hash, pdu, sizeof(*pdu)); @@ -2486,6 +2611,7 @@ static blk_status_t nvme_tcp_setup_cmd_pdu(struct nvme_ns *ns, if (req->curr_bio && req->data_len) nvme_tcp_init_iter(req, rq_data_dir(rq)); + req->offloaded = false; if (rq_data_dir(rq) == WRITE && req->data_len <= nvme_tcp_inline_data_size(queue)) req->pdu_len = req->data_len; From patchwork Thu Jul 22 11:02:56 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Boris Pismenny X-Patchwork-Id: 484322 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-20.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER, INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 50AEFC63793 for ; Thu, 22 Jul 2021 11:05:16 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 38E896121E for ; Thu, 22 Jul 2021 11:05:16 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231716AbhGVKYg (ORCPT ); Thu, 22 Jul 2021 06:24:36 -0400 Received: from mail-bn8nam12on2085.outbound.protection.outlook.com ([40.107.237.85]:43617 "EHLO NAM12-BN8-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S231717AbhGVKYQ (ORCPT ); Thu, 22 Jul 2021 06:24:16 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Mm0PAopKCgJk49IlCPscU7SNh/3mpWaUBk1fVoqeYhbkuSfHyIyQWTOmXKd/IiozaZBMbTxB07wIKvfxXpqqXGwmC6R6XiUwijLJPv+wu6hwDvmvq16LGuceOceGncUkCpQlU0x/isJp78KplyfrPdlRzjB5LJYbcnDP4nbeB6u90IL6zjdiAnikcHZh3t6JcpmmwLDns6tx8Bd4nyA9EcyTwI7URUNgRM6ANut4i1zySgCAdBbiheZo3mdiclC1EnwkPZ2fvIvZ+o1L08q+v+UO2+Dme2mAf+FlrgirCGjut/3Ac8/XyukOBcFewlLntyHaefWiVlB+aisNtwoUqg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=P7NCURwDQi4ArkZ7VF7g+H03CWQOyqQPqH7B9S++Aw4=; b=ZU7LwY1Lfrr1XZY082spDxYX6Xaj19wOdqKHD2PsLKwUxdddMm0FWj365LBqHlV8baSsfe32o1vxMghtgY8r4LXm+3iDH1mDXYvmZoMX50CQtwLJt3Po/I2DzLqmWsOvBydHhHSuiqONt1xO95eUcCJwnF69+m5VDNpLpuklt6kp2NBrMVVr9XJEOpEfvaH4KaLQsYeb26/1XvLkiPjsFa4tilAWG9BxlJFKBLmHTIDGcd6dYPR6UeJMfm3dnSOtV0Us9PWeEO71JxWTV0obK0JgEBJsJqUaTckR7Te3cgsXdeRkDKv7pPSxzlmY1ImCnHHbmt5VGANJyMU+x2gMPw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.34) smtp.rcpttodomain=marvell.com smtp.mailfrom=nvidia.com; dmarc=pass (p=quarantine sp=none pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=P7NCURwDQi4ArkZ7VF7g+H03CWQOyqQPqH7B9S++Aw4=; b=aUSHbtJbYdS02Wvavlav03qCrJ6TtkgUV5xYVRuK03UrUQKPh/y8aZuI0HCJSY1qhENSFXv+RDNQh4SRb4amnXkzHrn4HJLEYF8pmyOLBdtcCNPgjX7PV0UE8d77SKUZC0Dyf6STdd2rP021XdkVQBsJQdreRYDcKoc9hykYiyEeiz2yFiGRR8MZI8KqGzvxpYdFVSnNZL46p+XPrn0WalrZGhFaDhIMHJt2q74phZCl+w8wkKLL5RxbvEzzXV6wJT/E+pRdYZEqsK58P2aayhQOjLl3WAnZkh433H+ty76Y3Uw4Zm9WKWSMaBBP2T+YOklmtPlb3SThk/JKjQ4B7w== Received: from BN6PR19CA0068.namprd19.prod.outlook.com (2603:10b6:404:e3::30) by MWHPR12MB1326.namprd12.prod.outlook.com (2603:10b6:300:10::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4331.29; Thu, 22 Jul 2021 11:04:49 +0000 Received: from BN8NAM11FT029.eop-nam11.prod.protection.outlook.com (2603:10b6:404:e3:cafe::c1) by BN6PR19CA0068.outlook.office365.com (2603:10b6:404:e3::30) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4352.25 via Frontend Transport; Thu, 22 Jul 2021 11:04:48 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.34) smtp.mailfrom=nvidia.com; marvell.com; dkim=none (message not signed) header.d=none; marvell.com; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.34 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.34; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.34) by BN8NAM11FT029.mail.protection.outlook.com (10.13.177.68) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4352.24 via Frontend Transport; Thu, 22 Jul 2021 11:04:48 +0000 Received: from HQMAIL107.nvidia.com (172.20.187.13) by HQMAIL107.nvidia.com (172.20.187.13) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Thu, 22 Jul 2021 11:04:48 +0000 Received: from vdi.nvidia.com (172.20.187.5) by mail.nvidia.com (172.20.187.13) with Microsoft SMTP Server id 15.0.1497.2 via Frontend Transport; Thu, 22 Jul 2021 11:04:43 +0000 From: Boris Pismenny To: , , , , , , , , , , CC: , , , , , , Yoray Zack , Boris Pismenny , Ben Ben-Ishay , Or Gerlitz Subject: [PATCH v5 net-next 07/36] nvme-tcp: RX DDGST offload Date: Thu, 22 Jul 2021 14:02:56 +0300 Message-ID: <20210722110325.371-8-borisp@nvidia.com> X-Mailer: git-send-email 2.24.1 In-Reply-To: <20210722110325.371-1-borisp@nvidia.com> References: <20210722110325.371-1-borisp@nvidia.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: cc2619d2-7a9c-4a76-62a6-08d94d0085c2 X-MS-TrafficTypeDiagnostic: MWHPR12MB1326: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:4941; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 4jkCKAWKu3oZpQfBu8by86pwIouE3ajRg17HJQ41Kvp5sqPPL5Y3cVNTLFE7GtFevWzTiRvVW4Cc6NdsivZaSh/p2sfG7g6shoU6iVIV84rpdCu6wdDTAJh/m0KvHGQIkW37mmxn2lmLlYciNJiBDQ5BqOv9Rprw2BIqh4BXF9HDLpDr0zZUukRgsLHL6mGFy+lvnJC2y7JXbUV1hLyGQ/gIF75eA4xY9iTMnaDgWFucgZmRfuu/NjL13SggY114duiHdZiw8jY/cnOKKx/nSdJRVVa2bNEKkv9Gd0mamyoob76pPapYCatvHZCnjmMb8UlZ7LYkhiEOnhobjR1MZwS/C+FvgGyqeX/nSMsuax764QQ54GH1CS6J098C/L7jj76WknEybngEU6HzCjvSy/kk5hx22iXItZpMMoUwXpo7hstNGNBMsAAJt5TrA825Yb9OWtQAbWTOJ+i1Jv/XqH19fUBU29tmRwJvISIdcSns98UAKaGME+m4MSZDR25o0ftH+tDhTsoU/wF6tEzBH6aad4BtkHAbgzBP1Bmr+3/9DGT2wPbYnhDNwvK26Znk96LLn5SUQXIcP//Og/jFgyrdz4k+mX1DFkCfH/gYw9jnFx3XRZel7GEX7LCHmMbBEPEZlv4zXoYs3zYzyyB4wDPa25QVlA1+SEl0dzxgg7jsP1eSPwb2A1VdhzvAQrL2B3E7SY5rYHi7AMIW0Xs3+ZceuvXo93d2yGbb2EfENBM= X-Forefront-Antispam-Report: CIP:216.228.112.34; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:schybrid03.nvidia.com; CAT:NONE; SFS:(4636009)(46966006)(36840700001)(2616005)(2906002)(5660300002)(356005)(336012)(54906003)(1076003)(26005)(7696005)(86362001)(7416002)(508600001)(107886003)(36756003)(426003)(316002)(36906005)(921005)(4326008)(8676002)(47076005)(6666004)(70206006)(110136005)(8936002)(82310400003)(83380400001)(36860700001)(7636003)(186003)(70586007); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Jul 2021 11:04:48.7464 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: cc2619d2-7a9c-4a76-62a6-08d94d0085c2 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.112.34]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BN8NAM11FT029.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR12MB1326 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Yoray Zack Enable rx side of DDGST offload when supported. At the end of the capsule, check if all the skb bits are on, and if not recalculate the DDGST in SW and check it. We reworked the receive-side DDGST calculation to always run at the end, so as to keep a single flow for both offload and non-offload. This change simplifies the code, but it may degrade performance for non-offload DDGST calculation. Signed-off-by: Yoray Zack Signed-off-by: Boris Pismenny Signed-off-by: Ben Ben-Ishay Signed-off-by: Or Gerlitz --- drivers/nvme/host/tcp.c | 86 ++++++++++++++++++++++++++++++++++------- 1 file changed, 71 insertions(+), 15 deletions(-) diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c index 34982fb0c655..b23fdbb4fd8b 100644 --- a/drivers/nvme/host/tcp.c +++ b/drivers/nvme/host/tcp.c @@ -69,6 +69,7 @@ enum nvme_tcp_queue_flags { NVME_TCP_Q_LIVE = 1, NVME_TCP_Q_POLLING = 2, NVME_TCP_Q_OFF_DDP = 3, + NVME_TCP_Q_OFF_DDGST_RX = 4, }; enum nvme_tcp_recv_state { @@ -96,6 +97,7 @@ struct nvme_tcp_queue { size_t data_remaining; size_t ddgst_remaining; unsigned int nr_cqe; + bool ddgst_valid; /* send state */ struct nvme_tcp_request *request; @@ -230,6 +232,22 @@ static inline size_t nvme_tcp_pdu_last_send(struct nvme_tcp_request *req, return nvme_tcp_pdu_data_left(req) <= len; } +static inline bool nvme_tcp_ddp_ddgst_ok(struct nvme_tcp_queue *queue) +{ + return queue->ddgst_valid; +} + +static inline void nvme_tcp_ddp_ddgst_update(struct nvme_tcp_queue *queue, + struct sk_buff *skb) +{ + if (queue->ddgst_valid) +#ifdef CONFIG_ULP_DDP + queue->ddgst_valid = skb->ddp_crc; +#else + queue->ddgst_valid = false; +#endif +} + static int nvme_tcp_req_map_sg(struct nvme_tcp_request *req, struct request *rq) { int ret; @@ -243,6 +261,26 @@ static int nvme_tcp_req_map_sg(struct nvme_tcp_request *req, struct request *rq) return 0; } +static void nvme_tcp_ddp_ddgst_recalc(struct ahash_request *hash, + struct request *rq) +{ + struct nvme_tcp_request *req; + + if (!rq) + return; + + req = blk_mq_rq_to_pdu(rq); + + if (!req->offloaded && nvme_tcp_req_map_sg(req, rq)) + return; + + crypto_ahash_init(hash); + req->ddp.sg_table.sgl = req->ddp.first_sgl; + ahash_request_set_crypt(hash, req->ddp.sg_table.sgl, NULL, + le32_to_cpu(req->data_len)); + crypto_ahash_update(hash); +} + #ifdef CONFIG_ULP_DDP static bool nvme_tcp_resync_request(struct sock *sk, u32 seq, u32 flags); @@ -330,8 +368,10 @@ static int nvme_tcp_offload_socket(struct nvme_tcp_queue *queue) } inet_csk(queue->sock->sk)->icsk_ulp_ddp_ops = &nvme_tcp_ddp_ulp_ops; - if (netdev->features & NETIF_F_HW_ULP_DDP) + if (netdev->features & NETIF_F_HW_ULP_DDP) { set_bit(NVME_TCP_Q_OFF_DDP, &queue->flags); + set_bit(NVME_TCP_Q_OFF_DDGST_RX, &queue->flags); + } return ret; } @@ -346,6 +386,7 @@ static void nvme_tcp_unoffload_socket(struct nvme_tcp_queue *queue) } clear_bit(NVME_TCP_Q_OFF_DDP, &queue->flags); + clear_bit(NVME_TCP_Q_OFF_DDGST_RX, &queue->flags); netdev->ulp_ddp_ops->ulp_ddp_sk_del(netdev, queue->sock->sk); @@ -721,6 +762,7 @@ static void nvme_tcp_init_recv_ctx(struct nvme_tcp_queue *queue) queue->pdu_offset = 0; queue->data_remaining = -1; queue->ddgst_remaining = 0; + queue->ddgst_valid = true; } static void nvme_tcp_error_recovery(struct nvme_ctrl *ctrl) @@ -914,7 +956,8 @@ static int nvme_tcp_recv_pdu(struct nvme_tcp_queue *queue, struct sk_buff *skb, size_t rcv_len = min_t(size_t, *len, queue->pdu_remaining); int ret; - if (test_bit(NVME_TCP_Q_OFF_DDP, &queue->flags)) + if (test_bit(NVME_TCP_Q_OFF_DDP, &queue->flags) || + test_bit(NVME_TCP_Q_OFF_DDGST_RX, &queue->flags)) nvme_tcp_resync_response(queue, skb, *offset); ret = skb_copy_bits(skb, *offset, @@ -977,14 +1020,14 @@ static int nvme_tcp_consume_skb(struct nvme_tcp_queue *queue, struct sk_buff *sk #ifdef CONFIG_ULP_DDP if (test_bit(NVME_TCP_Q_OFF_DDP, &queue->flags)) { - if (queue->data_digest) + if (queue->data_digest && !test_bit(NVME_TCP_Q_OFF_DDGST_RX, &queue->flags)) ret = skb_ddp_copy_and_hash_datagram_iter(skb, *offset, iter, recv_len, queue->rcv_hash); else ret = skb_ddp_copy_datagram_iter(skb, *offset, iter, recv_len); } else { #endif - if (queue->data_digest) + if (queue->data_digest && !test_bit(NVME_TCP_Q_OFF_DDGST_RX, &queue->flags)) ret = skb_copy_and_hash_datagram_iter(skb, *offset, iter, recv_len, queue->rcv_hash); else @@ -1003,6 +1046,8 @@ static int nvme_tcp_recv_data(struct nvme_tcp_queue *queue, struct sk_buff *skb, struct nvme_tcp_request *req; struct request *rq; + if (queue->data_digest && test_bit(NVME_TCP_Q_OFF_DDGST_RX, &queue->flags)) + nvme_tcp_ddp_ddgst_update(queue, skb); rq = blk_mq_tag_to_rq(nvme_tcp_tagset(queue), pdu->command_id); if (!rq) { dev_err(queue->ctrl->ctrl.device, @@ -1055,7 +1100,6 @@ static int nvme_tcp_recv_data(struct nvme_tcp_queue *queue, struct sk_buff *skb, if (!queue->data_remaining) { if (queue->data_digest) { - nvme_tcp_ddgst_final(queue->rcv_hash, &queue->exp_ddgst); queue->ddgst_remaining = NVME_TCP_DIGEST_LENGTH; } else { if (pdu->hdr.flags & NVME_TCP_F_DATA_SUCCESS) { @@ -1076,8 +1120,12 @@ static int nvme_tcp_recv_ddgst(struct nvme_tcp_queue *queue, char *ddgst = (char *)&queue->recv_ddgst; size_t recv_len = min_t(size_t, *len, queue->ddgst_remaining); off_t off = NVME_TCP_DIGEST_LENGTH - queue->ddgst_remaining; + bool offload_fail, offload_en; + struct request *rq = NULL; int ret; + if (test_bit(NVME_TCP_Q_OFF_DDGST_RX, &queue->flags)) + nvme_tcp_ddp_ddgst_update(queue, skb); ret = skb_copy_bits(skb, *offset, &ddgst[off], recv_len); if (unlikely(ret)) return ret; @@ -1088,18 +1136,25 @@ static int nvme_tcp_recv_ddgst(struct nvme_tcp_queue *queue, if (queue->ddgst_remaining) return 0; - if (queue->recv_ddgst != queue->exp_ddgst) { - dev_err(queue->ctrl->ctrl.device, - "data digest error: recv %#x expected %#x\n", - le32_to_cpu(queue->recv_ddgst), - le32_to_cpu(queue->exp_ddgst)); - return -EIO; + rq = blk_mq_tag_to_rq(nvme_tcp_tagset(queue), pdu->command_id); + + offload_fail = !nvme_tcp_ddp_ddgst_ok(queue); + offload_en = test_bit(NVME_TCP_Q_OFF_DDGST_RX, &queue->flags); + if (!offload_en || offload_fail) { + if (offload_en && offload_fail) // software-fallback + nvme_tcp_ddp_ddgst_recalc(queue->rcv_hash, rq); + + nvme_tcp_ddgst_final(queue->rcv_hash, &queue->exp_ddgst); + if (queue->recv_ddgst != queue->exp_ddgst) { + dev_err(queue->ctrl->ctrl.device, + "data digest error: recv %#x expected %#x\n", + le32_to_cpu(queue->recv_ddgst), + le32_to_cpu(queue->exp_ddgst)); + return -EIO; + } } if (pdu->hdr.flags & NVME_TCP_F_DATA_SUCCESS) { - struct request *rq = blk_mq_tag_to_rq(nvme_tcp_tagset(queue), - pdu->command_id); - nvme_tcp_end_request(rq, NVME_SC_SUCCESS); queue->nr_cqe++; } @@ -1851,7 +1906,8 @@ static void __nvme_tcp_stop_queue(struct nvme_tcp_queue *queue) nvme_tcp_restore_sock_calls(queue); cancel_work_sync(&queue->io_work); - if (test_bit(NVME_TCP_Q_OFF_DDP, &queue->flags)) + if (test_bit(NVME_TCP_Q_OFF_DDP, &queue->flags) || + test_bit(NVME_TCP_Q_OFF_DDGST_RX, &queue->flags)) nvme_tcp_unoffload_socket(queue); } From patchwork Thu Jul 22 11:02:57 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Boris Pismenny X-Patchwork-Id: 485109 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-20.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER, INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 49D86C63797 for ; Thu, 22 Jul 2021 11:05:08 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 301A76120D for ; Thu, 22 Jul 2021 11:05:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231740AbhGVKYc (ORCPT ); Thu, 22 Jul 2021 06:24:32 -0400 Received: from mail-bn7nam10on2079.outbound.protection.outlook.com ([40.107.92.79]:31840 "EHLO NAM10-BN7-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S231728AbhGVKYU (ORCPT ); Thu, 22 Jul 2021 06:24:20 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=lQ9VnAcjc7Le7SVT3wP8L5PcvxTHvRMj9g6SQ3TR2wAFYHdr82Eso94U0v1Ro2vCWHHUWrl9wp3TWC7CbEWOJpm5fjBbTgI+LnqTMF4TVIyukwgUdDkPlMYV0XfhTiHCE1A55gQPeCwVnGL2rbmAl27XNrHHnrdytj13Ordnn5QW4dbZp/Jd7XCJWBeNC14vaSNW908nooc+PGCY1aM8GX781WJ2/MpaSF3/xsu6OVdJbhd3J52/WMWGM56BX3b1q4lu+J8sTcZYWc8IJR2tgIgDjivPopEW4H29xMu8RgHNnDMSvwex0NVFoD3RowYfyt3/WJ2LIFIRhNSQvHCM9Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=pOFQTho0/UFbIeyik9Lh2h8iIZLpT3ZqHVJkQmyPnjE=; b=njSUriBVG+cUBww/w6xZ8Q5uLXLlhnI675EA/TwkvDu4phxR/uQ3kClTzCx+FWwcbNhH+DUxQU3OCdPm60MKo8734kisAMh0I8BHNt1y/TK2hdKlEPHoPWKMJGgw5giQZ4eh8CA7Qfbc1gq/LMBvGIKNDehqmCDdAfqeeTYsYw1ry+LHgrRjlPsJIPZQeEYMT6Jn1R+w1E7q8gTHev8Rj4HuoF/7h4pUPQ+0DpZ6TUvvPjCh3573kynCYVdYU4dWZuNUspAODEuLGk1bQDkvay2dIg4Hvq9AVgrUBXZH+SVvE3WVzO1nDqWhrmjz5kAXyRKJWC8cQtCDw0nCZ6yF/Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.35) smtp.rcpttodomain=marvell.com smtp.mailfrom=nvidia.com; dmarc=pass (p=quarantine sp=none pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=pOFQTho0/UFbIeyik9Lh2h8iIZLpT3ZqHVJkQmyPnjE=; b=O3CDVlCvlUeiwhE2loHhavwwoeEjwkZaFdmwvhi+qzqR14ApGjppYaFjhA96G++iAuCM2iMUHJjjjfhJgMK55E6BXvKSU4+jo50/aFopgQOdZZ4n6ENdwDoDVkeomNmoNM0loVaX7Psf7iHvnLrLRYj8LLhqto3d96bWRZ15zAJsBx16QCKAIkK65Z0liGtsao3JWuuoDZ0sNIXwoOUD1BeNecKoRlDNRzqzR447nJ2QqQA5oJLlxE62+rWmJX+B24Dr8Wm/nPGEyJGX2VXMm5YYr0NE6DfWw51AoCQMuFHuQETSiOqMk4oPkDJnwoqhvUV63NEpCFi/leiW5MYplQ== Received: from DM6PR11CA0062.namprd11.prod.outlook.com (2603:10b6:5:14c::39) by MN2PR12MB3773.namprd12.prod.outlook.com (2603:10b6:208:164::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4331.26; Thu, 22 Jul 2021 11:04:54 +0000 Received: from DM6NAM11FT030.eop-nam11.prod.protection.outlook.com (2603:10b6:5:14c:cafe::a8) by DM6PR11CA0062.outlook.office365.com (2603:10b6:5:14c::39) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4352.26 via Frontend Transport; Thu, 22 Jul 2021 11:04:54 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.35) smtp.mailfrom=nvidia.com; marvell.com; dkim=none (message not signed) header.d=none; marvell.com; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.35 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.35; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.35) by DM6NAM11FT030.mail.protection.outlook.com (10.13.172.146) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4352.24 via Frontend Transport; Thu, 22 Jul 2021 11:04:53 +0000 Received: from HQMAIL107.nvidia.com (172.20.187.13) by HQMAIL111.nvidia.com (172.20.187.18) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Thu, 22 Jul 2021 11:04:52 +0000 Received: from vdi.nvidia.com (172.20.187.5) by mail.nvidia.com (172.20.187.13) with Microsoft SMTP Server id 15.0.1497.2 via Frontend Transport; Thu, 22 Jul 2021 11:04:48 +0000 From: Boris Pismenny To: , , , , , , , , , , CC: , , , , , , Or Gerlitz , Boris Pismenny , Ben Ben-Ishay , Yoray Zack Subject: [PATCH v5 net-next 08/36] nvme-tcp: Deal with netdevice DOWN events Date: Thu, 22 Jul 2021 14:02:57 +0300 Message-ID: <20210722110325.371-9-borisp@nvidia.com> X-Mailer: git-send-email 2.24.1 In-Reply-To: <20210722110325.371-1-borisp@nvidia.com> References: <20210722110325.371-1-borisp@nvidia.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: de370a7b-bcab-4044-804e-08d94d0088b9 X-MS-TrafficTypeDiagnostic: MN2PR12MB3773: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:3276; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: kNfDs4fpORv5TnLYYXtq3JJUpnBgCGyGRqg1X8fvpA/othpzbl6i8LYj1XcVdYIUI/gk9cKJ3xwZ7+dABvvPCjRUynmDRjSYcrmc/baEIm3ksSd0ifrGywOrQI5nLDmuN5j5C00eSKG79O/h37uAMTNF5f4XymTJpqW1fRNsIXJeIT/omh3DCyWj+qwwU6Q6e/M7heQnpdwzHCJMi2DngGvuKpnvPvBQTg+F9+XToXozCNqJFHepYHnTQnV3k15iU0DxgKJcYmDax4BMUPHiOQpAi0dHNbsiCUqBT8eu9hcseC+1EDpn4yl+kAqMi67SKTKDGyMW0oLLSCrAZx+IAHguPik76BtryPwVj1+vEPQ2HpErSIHkWbfL9yxwQtV+uPqRGtplAF9Q7QeMQVbLclAe0emtrv40GnWQUE98uoCOz9ZYybe4Ui+kaQH5g4jtLJCKjfyBQn1wyh2fYtXtUiXBOxAP5+YxN0Oose5o4qv94HZulI+oaGe9pQg/irsVNAgKAXFDY7P1R7uoCE4qLbHyh8cwA0rTBkWHTZLioU3gnfNIJSwHmqxqWO1oc+ZBY8pyqd3Tmvf6H/RtCFy8/ph5jm9OGP9RW5CZJZWiTxBwXRcCWjqWeVL03wrrsv6oF8zfcAddVgaWGDlQbSfzRLF2U9czgDcH3d7pD1fvETECI7yyRqAN3kVqpgtHDP2jGPuWy6NFwCVkGg7TNy3Drw== X-Forefront-Antispam-Report: CIP:216.228.112.35; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:schybrid04.nvidia.com; CAT:NONE; SFS:(4636009)(396003)(376002)(136003)(39860400002)(346002)(46966006)(36840700001)(86362001)(7696005)(26005)(921005)(2616005)(6666004)(2906002)(8936002)(5660300002)(36860700001)(426003)(82740400003)(70586007)(36906005)(70206006)(7636003)(110136005)(336012)(107886003)(478600001)(82310400003)(47076005)(83380400001)(316002)(186003)(36756003)(1076003)(54906003)(4326008)(7416002)(8676002)(356005); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Jul 2021 11:04:53.7563 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: de370a7b-bcab-4044-804e-08d94d0088b9 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.112.35]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DM6NAM11FT030.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR12MB3773 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Or Gerlitz For ddp setup/teardown and resync, the offloading logic uses HW resources at the NIC driver such as SQ and CQ. These resources are destroyed when the netdevice does down and hence we must stop using them before the NIC driver destroys them. Use netdevice notifier for that matter -- offloaded connections are stopped before the stack continues to call the NIC driver close ndo. We use the existing recovery flow which has the advantage of resuming the offload once the connection is re-set. This also buys us proper handling for the UNREGISTER event b/c our offloading starts in the UP state, and down is always there between up to unregister. Signed-off-by: Or Gerlitz Signed-off-by: Boris Pismenny Signed-off-by: Ben Ben-Ishay Signed-off-by: Yoray Zack --- drivers/nvme/host/tcp.c | 39 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 39 insertions(+) diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c index b23fdbb4fd8b..b338cd2d9f65 100644 --- a/drivers/nvme/host/tcp.c +++ b/drivers/nvme/host/tcp.c @@ -147,6 +147,7 @@ struct nvme_tcp_ctrl { static LIST_HEAD(nvme_tcp_ctrl_list); static DEFINE_MUTEX(nvme_tcp_ctrl_mutex); +static struct notifier_block nvme_tcp_netdevice_nb; static struct workqueue_struct *nvme_tcp_wq; static const struct blk_mq_ops nvme_tcp_mq_ops; static const struct blk_mq_ops nvme_tcp_admin_mq_ops; @@ -2948,6 +2949,30 @@ static struct nvme_ctrl *nvme_tcp_create_ctrl(struct device *dev, return ERR_PTR(ret); } +static int nvme_tcp_netdev_event(struct notifier_block *this, + unsigned long event, void *ptr) +{ + struct net_device *ndev = netdev_notifier_info_to_dev(ptr); + struct nvme_tcp_ctrl *ctrl; + + switch (event) { + case NETDEV_GOING_DOWN: + mutex_lock(&nvme_tcp_ctrl_mutex); + list_for_each_entry(ctrl, &nvme_tcp_ctrl_list, list) { + if (ndev != ctrl->offloading_netdev) + continue; + nvme_tcp_error_recovery(&ctrl->ctrl); + } + mutex_unlock(&nvme_tcp_ctrl_mutex); + flush_workqueue(nvme_reset_wq); + /* + * The associated controllers teardown has completed, ddp contexts + * were also torn down so we should be safe to continue... + */ + } + return NOTIFY_DONE; +} + static struct nvmf_transport_ops nvme_tcp_transport = { .name = "tcp", .module = THIS_MODULE, @@ -2962,13 +2987,26 @@ static struct nvmf_transport_ops nvme_tcp_transport = { static int __init nvme_tcp_init_module(void) { + int ret; + nvme_tcp_wq = alloc_workqueue("nvme_tcp_wq", WQ_MEM_RECLAIM | WQ_HIGHPRI, 0); if (!nvme_tcp_wq) return -ENOMEM; + nvme_tcp_netdevice_nb.notifier_call = nvme_tcp_netdev_event; + ret = register_netdevice_notifier(&nvme_tcp_netdevice_nb); + if (ret) { + pr_err("failed to register netdev notifier\n"); + goto out_err_reg_notifier; + } + nvmf_register_transport(&nvme_tcp_transport); return 0; + +out_err_reg_notifier: + destroy_workqueue(nvme_tcp_wq); + return ret; } static void __exit nvme_tcp_cleanup_module(void) @@ -2976,6 +3014,7 @@ static void __exit nvme_tcp_cleanup_module(void) struct nvme_tcp_ctrl *ctrl; nvmf_unregister_transport(&nvme_tcp_transport); + unregister_netdevice_notifier(&nvme_tcp_netdevice_nb); mutex_lock(&nvme_tcp_ctrl_mutex); list_for_each_entry(ctrl, &nvme_tcp_ctrl_list, list) From patchwork Thu Jul 22 11:02:58 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Boris Pismenny X-Patchwork-Id: 485107 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-20.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER, INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 28E09C63797 for ; Thu, 22 Jul 2021 11:05:25 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 13B1160725 for ; Thu, 22 Jul 2021 11:05:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231765AbhGVKYt (ORCPT ); Thu, 22 Jul 2021 06:24:49 -0400 Received: from mail-bn7nam10on2053.outbound.protection.outlook.com ([40.107.92.53]:34273 "EHLO NAM10-BN7-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S231698AbhGVKYZ (ORCPT ); Thu, 22 Jul 2021 06:24:25 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=L61T+itCnm7mjNpuuAwvQvEQYJt/ulV5R7Ax3TLicmpbdDq8CtuL9Z3910YO+WuOY+tgCgNnAkhIzD5yoB/KBrGRfNrX0c9AWfH4tSqiB4NvjyLUCED3fvNAQb4JCxhdvJNTUtYR7po9cN3UQLHMmM+q4GWIemYEfecZQi5VHwfpDQtoxPIpegXpxzrjgaqiHrfJyAQh6RKrKO5S0zCRkmi1KImXj06sh4mRvRCs9jmuW2dOTEWBE7wu5vDauRq2AjPuca/9RJN1l9taYDzTdFr460AKiUw7ONhzV7a4o7kpttkU+cwNm1Y1NW4SxUla8GxZeEj6e+Pj9lmG8JrbYQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=2/b1P+mmdEkY/2jjay+h1NlMcdiO1zcXNI0ggrM6BnA=; b=D6EKjc6j3DtAjML2F6p7tKZGs/KtmOf/9saa5dqwbt0CVofRAWvy3MY6MTVCDYzvdQgn4ED74rp+97VZv5wRNfZInOT3HwBx+6/ZeQhHb5AFQsDWa5DzeNF+VcTmCnb5G2Km+f1XPUve0KDy7Fw1zr6FTUagxfwf95slYeDxL5rDQkaGfx43r8475uFZJR2z+BytdQZSii2n/R1diMBvXgmevLbBdRcNOc/rZ9yD6x5tuZmgjVC4FmcnIxw4zRxBt1eGFXh/oLcFN/+v1myoZfcodQu6LVplIcZjuxVx0YngS41VLWQAddl4wsvGayez/0dtU4h/zg3jK3ef64cA9Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.34) smtp.rcpttodomain=marvell.com smtp.mailfrom=nvidia.com; dmarc=pass (p=quarantine sp=none pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=2/b1P+mmdEkY/2jjay+h1NlMcdiO1zcXNI0ggrM6BnA=; b=HOubwWC5RTlXZpxtXRQEcfFa/smoYgNE5oCftcAn8e/ECmKrC812Hyzz5Nsl2m/PqJw+hU9aCddmNCrDIyxIPWICIFyUu28aWMdc1Xyx397/b+e9EfN4Hz6BexIs8IwS4HwsS/1xE5oKF4wYBw8vrIVGHUFpDJSonHtcKFzguAlztJwbLB25h66oWRLGqYTbfNN7Mcqpr//jHmFB1xZbmj4k7mdybRs2RHW3hhjtBskrcuGyLDEGwoetseiwZGKHXhacqQCKJurmQD2ol0A3oVQ1ZIUd9GOfdHJE9XF9urymw/NHm3DMhecSuMWD7K1RMDrKAsSeb866QvFDMoC3Hw== Received: from BN0PR03CA0002.namprd03.prod.outlook.com (2603:10b6:408:e6::7) by SN1PR12MB2477.namprd12.prod.outlook.com (2603:10b6:802:28::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4331.26; Thu, 22 Jul 2021 11:04:58 +0000 Received: from BN8NAM11FT004.eop-nam11.prod.protection.outlook.com (2603:10b6:408:e6:cafe::a0) by BN0PR03CA0002.outlook.office365.com (2603:10b6:408:e6::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4352.25 via Frontend Transport; Thu, 22 Jul 2021 11:04:58 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.34) smtp.mailfrom=nvidia.com; marvell.com; dkim=none (message not signed) header.d=none; marvell.com; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.34 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.34; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.34) by BN8NAM11FT004.mail.protection.outlook.com (10.13.176.164) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4352.24 via Frontend Transport; Thu, 22 Jul 2021 11:04:58 +0000 Received: from HQMAIL107.nvidia.com (172.20.187.13) by HQMAIL107.nvidia.com (172.20.187.13) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Thu, 22 Jul 2021 11:04:57 +0000 Received: from vdi.nvidia.com (172.20.187.5) by mail.nvidia.com (172.20.187.13) with Microsoft SMTP Server id 15.0.1497.2 via Frontend Transport; Thu, 22 Jul 2021 11:04:53 +0000 From: Boris Pismenny To: , , , , , , , , , , CC: , , , , , , Boris Pismenny , Or Gerlitz , Yoray Zack Subject: [PATCH v5 net-next 09/36] net/mlx5: Header file changes for nvme-tcp offload Date: Thu, 22 Jul 2021 14:02:58 +0300 Message-ID: <20210722110325.371-10-borisp@nvidia.com> X-Mailer: git-send-email 2.24.1 In-Reply-To: <20210722110325.371-1-borisp@nvidia.com> References: <20210722110325.371-1-borisp@nvidia.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 9b5a8e08-5a0a-4590-05a0-08d94d008b8f X-MS-TrafficTypeDiagnostic: SN1PR12MB2477: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:3968; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 8TIr0Kkgv11QmEiDfcCwJWLigkBYgL93goRAYxiMIBv4R2lsW0QdBYrqHHBS5fyWYqGvGg1K2SMfXLLVhsNJy768Xp9X+vcBdhocG3WQmifg5G+Y2GUXrSTXN4xylYg1HaRtAHL42K7fDxaw4RT9g+DXJdeag/rjGGfEwf4/4zq3i59lW8cj5wyafn3PaePxxnrIp64RKX6cOVKzLA6s5gBXeskkDHjbkIiLrvSW/8/kLblI73TEDX7yDzCnLWL1yNERXCvS01bmpUYYscC1SbLj5xoPJOXoWMuJacOUdRnXThWq0mhkgvi0lMS+fI05VKIRrLYbdjKPBTQC6/M6qF9xY7RYrWyk/NX5XREqG1IW8EGhr4puapoz8rzJb2VBzVe0tdt8ZEvb7zSqZRSXexvc7XGOhpIP26WhgdL+8EpoFpRRau7OhoNE02/II/QIzbvnU4SD8WhZuAoagCGgAYJ5MD6nunFuCE5VHiZQa7qnN/j/eolP5KhxS+1lfHIRnjpZMnY+K12ZxMTMpUHcQRzowaARzXVNbViRmCGP0AOYgOUI5oMLYICIGTEFvNxG1EXsBUJ2IerpEs3AtIfkzQMLrge8CYqZSJ4IRoFalQx8BhCi5gShyjoYfl3j9tYM569Jg1HwbjE4YRpdpz/wuDDu486lse1DszBb+XfCrV6HfldvOF5+0N4zth3Pcz1/h2vjLDT2TkUFCou3FHED98gOz3pOHidff60KBcFoyBE= X-Forefront-Antispam-Report: CIP:216.228.112.34; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:schybrid03.nvidia.com; CAT:NONE; SFS:(4636009)(136003)(376002)(396003)(39860400002)(346002)(36840700001)(46966006)(5660300002)(36906005)(54906003)(82310400003)(426003)(86362001)(356005)(110136005)(7696005)(316002)(1076003)(36860700001)(8936002)(47076005)(336012)(2616005)(82740400003)(107886003)(70586007)(70206006)(36756003)(8676002)(7636003)(186003)(921005)(26005)(478600001)(4326008)(83380400001)(2906002)(7416002); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Jul 2021 11:04:58.4691 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 9b5a8e08-5a0a-4590-05a0-08d94d008b8f X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.112.34]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BN8NAM11FT004.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN1PR12MB2477 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Ben Ben-ishay Add the necessary infrastructure for NVMEoTCP offload: - Add nvmeocp_en + nvmeotcp_crc_en bit to the TIR for identify NVMEoTCP offload flow And tag_buffer_id that will be used by the connected nvmeotcp_queues - Add new CQE field that will be used to pass scattered data information to SW - Add new capability to HCA_CAP that represnts the NVMEoTCP offload ability Signed-off-by: Ben Ben-ishay Signed-off-by: Boris Pismenny Signed-off-by: Or Gerlitz Signed-off-by: Yoray Zack --- include/linux/mlx5/device.h | 8 +++ include/linux/mlx5/mlx5_ifc.h | 101 +++++++++++++++++++++++++++++++++- include/linux/mlx5/qp.h | 1 + 3 files changed, 107 insertions(+), 3 deletions(-) diff --git a/include/linux/mlx5/device.h b/include/linux/mlx5/device.h index 0025913505ab..a42e47f91327 100644 --- a/include/linux/mlx5/device.h +++ b/include/linux/mlx5/device.h @@ -263,6 +263,7 @@ enum { enum { MLX5_MKEY_MASK_LEN = 1ull << 0, MLX5_MKEY_MASK_PAGE_SIZE = 1ull << 1, + MLX5_MKEY_MASK_XLT_OCT_SIZE = 1ull << 2, MLX5_MKEY_MASK_START_ADDR = 1ull << 6, MLX5_MKEY_MASK_PD = 1ull << 7, MLX5_MKEY_MASK_EN_RINVAL = 1ull << 8, @@ -1179,6 +1180,7 @@ enum mlx5_cap_type { MLX5_CAP_VDPA_EMULATION = 0x13, MLX5_CAP_DEV_EVENT = 0x14, MLX5_CAP_IPSEC, + MLX5_CAP_DEV_NVMEOTCP = 0x19, MLX5_CAP_GENERAL_2 = 0x20, /* NUM OF CAP Types */ MLX5_CAP_NUM @@ -1409,6 +1411,12 @@ enum mlx5_qcam_feature_groups { #define MLX5_CAP_IPSEC(mdev, cap)\ MLX5_GET(ipsec_cap, (mdev)->caps.hca_cur[MLX5_CAP_IPSEC], cap) +#define MLX5_CAP_DEV_NVMEOTCP(mdev, cap)\ + MLX5_GET(nvmeotcp_cap, mdev->caps.hca_cur[MLX5_CAP_DEV_NVMEOTCP], cap) + +#define MLX5_CAP64_NVMEOTCP(mdev, cap)\ + MLX5_GET64(nvmeotcp_cap, mdev->caps.hca_cur[MLX5_CAP_DEV_NVMEOTCP], cap) + enum { MLX5_CMD_STAT_OK = 0x0, MLX5_CMD_STAT_INT_ERR = 0x1, diff --git a/include/linux/mlx5/mlx5_ifc.h b/include/linux/mlx5/mlx5_ifc.h index e32a0d61929b..f0310c24f408 100644 --- a/include/linux/mlx5/mlx5_ifc.h +++ b/include/linux/mlx5/mlx5_ifc.h @@ -1332,7 +1332,9 @@ struct mlx5_ifc_cmd_hca_cap_bits { u8 event_cap[0x1]; u8 reserved_at_91[0x2]; u8 isolate_vl_tc_new[0x1]; - u8 reserved_at_94[0x4]; + u8 reserved_at_94[0x2]; + u8 nvmeotcp[0x1]; + u8 reserved_at_97[0x1]; u8 prio_tag_required[0x1]; u8 reserved_at_99[0x2]; u8 log_max_qp[0x5]; @@ -3118,6 +3120,21 @@ struct mlx5_ifc_roce_addr_layout_bits { u8 reserved_at_e0[0x20]; }; +struct mlx5_ifc_nvmeotcp_cap_bits { + u8 zerocopy[0x1]; + u8 crc_rx[0x1]; + u8 crc_tx[0x1]; + u8 reserved_at_3[0x15]; + u8 version[0x8]; + + u8 reserved_at_20[0x13]; + u8 log_max_nvmeotcp_tag_buffer_table[0x5]; + u8 reserved_at_38[0x3]; + u8 log_max_nvmeotcp_tag_buffer_size[0x5]; + + u8 reserved_at_40[0x7c0]; +}; + union mlx5_ifc_hca_cap_union_bits { struct mlx5_ifc_cmd_hca_cap_bits cmd_hca_cap; struct mlx5_ifc_cmd_hca_cap_2_bits cmd_hca_cap_2; @@ -3135,6 +3152,7 @@ union mlx5_ifc_hca_cap_union_bits { struct mlx5_ifc_tls_cap_bits tls_cap; struct mlx5_ifc_device_mem_cap_bits device_mem_cap; struct mlx5_ifc_virtio_emulation_cap_bits virtio_emulation_cap; + struct mlx5_ifc_nvmeotcp_cap_bits nvmeotcp_cap; u8 reserved_at_0[0x8000]; }; @@ -3329,7 +3347,9 @@ struct mlx5_ifc_tirc_bits { u8 disp_type[0x4]; u8 tls_en[0x1]; - u8 reserved_at_25[0x1b]; + u8 nvmeotcp_zero_copy_en[0x1]; + u8 nvmeotcp_crc_en[0x1]; + u8 reserved_at_27[0x19]; u8 reserved_at_40[0x40]; @@ -3360,7 +3380,8 @@ struct mlx5_ifc_tirc_bits { struct mlx5_ifc_rx_hash_field_select_bits rx_hash_field_selector_inner; - u8 reserved_at_2c0[0x4c0]; + u8 nvmeotcp_tag_buffer_table_id[0x20]; + u8 reserved_at_2e0[0x4a0]; }; enum { @@ -10976,12 +10997,14 @@ enum { MLX5_HCA_CAP_GENERAL_OBJECT_TYPES_ENCRYPTION_KEY = BIT_ULL(0xc), MLX5_HCA_CAP_GENERAL_OBJECT_TYPES_IPSEC = BIT_ULL(0x13), MLX5_HCA_CAP_GENERAL_OBJECT_TYPES_SAMPLER = BIT_ULL(0x20), + MLX5_HCA_CAP_GENERAL_OBJECT_TYPES_NVMEOTCP_TAG_BUFFER_TABLE = BIT_ULL(0x21), }; enum { MLX5_GENERAL_OBJECT_TYPES_ENCRYPTION_KEY = 0xc, MLX5_GENERAL_OBJECT_TYPES_IPSEC = 0x13, MLX5_GENERAL_OBJECT_TYPES_SAMPLER = 0x20, + MLX5_GENERAL_OBJECT_TYPES_NVMEOTCP_TAG_BUFFER_TABLE = 0x21 }; enum { @@ -11088,6 +11111,20 @@ struct mlx5_ifc_query_sampler_obj_out_bits { struct mlx5_ifc_sampler_obj_bits sampler_object; }; +struct mlx5_ifc_nvmeotcp_tag_buf_table_obj_bits { + u8 modify_field_select[0x40]; + + u8 reserved_at_20[0x20]; + + u8 reserved_at_40[0x1b]; + u8 log_tag_buffer_table_size[0x5]; +}; + +struct mlx5_ifc_create_nvmeotcp_tag_buf_table_in_bits { + struct mlx5_ifc_general_obj_in_cmd_hdr_bits general_obj_in_cmd_hdr; + struct mlx5_ifc_nvmeotcp_tag_buf_table_obj_bits nvmeotcp_tag_buf_table_obj; +}; + enum { MLX5_GENERAL_OBJECT_TYPE_ENCRYPTION_KEY_KEY_SIZE_128 = 0x0, MLX5_GENERAL_OBJECT_TYPE_ENCRYPTION_KEY_KEY_SIZE_256 = 0x1, @@ -11098,6 +11135,18 @@ enum { MLX5_GENERAL_OBJECT_TYPE_ENCRYPTION_KEY_TYPE_IPSEC = 0x2, }; +enum { + MLX5_TRANSPORT_STATIC_PARAMS_ACC_TYPE_XTS = 0x0, + MLX5_TRANSPORT_STATIC_PARAMS_ACC_TYPE_TLS = 0x1, + MLX5_TRANSPORT_STATIC_PARAMS_ACC_TYPE_NVMETCP = 0x2, + MLX5_TRANSPORT_STATIC_PARAMS_ACC_TYPE_NVMETCP_WITH_TLS = 0x3, +}; + +enum { + MLX5_TRANSPORT_STATIC_PARAMS_TI_INITIATOR = 0x0, + MLX5_TRANSPORT_STATIC_PARAMS_TI_TARGET = 0x1, +}; + struct mlx5_ifc_tls_static_params_bits { u8 const_2[0x2]; u8 tls_version[0x4]; @@ -11138,4 +11187,50 @@ enum { MLX5_MTT_PERM_RW = MLX5_MTT_PERM_READ | MLX5_MTT_PERM_WRITE, }; +struct mlx5_ifc_nvmeotcp_progress_params_bits { + u8 next_pdu_tcp_sn[0x20]; + + u8 hw_resync_tcp_sn[0x20]; + + u8 pdu_tracker_state[0x2]; + u8 offloading_state[0x2]; + u8 reserved_at_64[0xc]; + u8 cccid_ttag[0x10]; +}; + +struct mlx5_ifc_transport_static_params_bits { + u8 const_2[0x2]; + u8 tls_version[0x4]; + u8 const_1[0x2]; + u8 reserved_at_8[0x14]; + u8 acc_type[0x4]; + + u8 reserved_at_20[0x20]; + + u8 initial_record_number[0x40]; + + u8 resync_tcp_sn[0x20]; + + u8 gcm_iv[0x20]; + + u8 implicit_iv[0x40]; + + u8 reserved_at_100[0x8]; + u8 dek_index[0x18]; + + u8 reserved_at_120[0x14]; + u8 const1[0x1]; + u8 ti[0x1]; + u8 zero_copy_en[0x1]; + u8 ddgst_offload_en[0x1]; + u8 hdgst_offload_en[0x1]; + u8 ddgst_en[0x1]; + u8 hddgst_en[0x1]; + u8 pda[0x5]; + + u8 nvme_resync_tcp_sn[0x20]; + + u8 reserved_at_160[0xa0]; +}; + #endif /* MLX5_IFC_H */ diff --git a/include/linux/mlx5/qp.h b/include/linux/mlx5/qp.h index b7deb790f257..dfd744a564d5 100644 --- a/include/linux/mlx5/qp.h +++ b/include/linux/mlx5/qp.h @@ -220,6 +220,7 @@ struct mlx5_wqe_ctrl_seg { #define MLX5_WQE_CTRL_OPCODE_MASK 0xff #define MLX5_WQE_CTRL_WQE_INDEX_MASK 0x00ffff00 #define MLX5_WQE_CTRL_WQE_INDEX_SHIFT 8 +#define MLX5_WQE_CTRL_TIR_TIS_INDEX_SHIFT 8 enum { MLX5_ETH_WQE_L3_INNER_CSUM = 1 << 4, From patchwork Thu Jul 22 11:02:59 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Boris Pismenny X-Patchwork-Id: 485108 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-20.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER, INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E8FF5C63793 for ; Thu, 22 Jul 2021 11:05:21 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id C7A146120D for ; Thu, 22 Jul 2021 11:05:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231648AbhGVKYo (ORCPT ); Thu, 22 Jul 2021 06:24:44 -0400 Received: from mail-dm6nam10on2064.outbound.protection.outlook.com ([40.107.93.64]:19866 "EHLO NAM10-DM6-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S231724AbhGVKY3 (ORCPT ); Thu, 22 Jul 2021 06:24:29 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=iH2obv2QdaLnrC4Y4W4Gpx1THhSXDWrky884BohjaWOYvXZq/DYJ0tc0EEXUPEBKpgyWZXFp61PKLAxN1teb2txmIzHwkD/kLMFbsZpd7eERpLSPWZhfjturKSR27oIQRDbRrJ+ZfyUO1iWUZscZBCut5BqZ5uLKe/AFPGbzs84Hx4Yw8TAMYHpL1RiumOsy66OotDtA3CyK59boYkbu4sezPd7eDVGvQM7D+QAS3oQaDO+1aBt0JDiHcjJnOwupPJprnHavwD2OEs/HoZ25QItkJjrZH71X3KGHXTZwEKQne8EQg8lRl9NG1qWV6z36y0p6pVgbkQUxm/M8Eh2Xrg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=8TVpBQt+V6f6hp9z2D2qjnhFtrl+F5aIzj80UUbtCE8=; b=jREwO5MxuN3QolWJ5IdAOaMaicol0Zirn8wOPdxurzPdUoDdzYdHfE4XPObtUCKY9Leo0m/SAcrzaYOuLas6rsVVYBHQmA0AUEl2QRcWmZ5syIY5s3tWl+WunIfdw0a3vBfeSMgutTO7SNgyRDBv4aSwoMp6Tz3V4gZ7d1GbshjDTW10NIaH0vPg0n9JUj+3Yk/WBWqTB8xw7bJzcHFyMuHPs//oe0eCyeKs7zbBu+7AnzxUainmI7NijG3uy1jXjcFGD5qb6uYOQTyxJ9C7qsfQK5R9qYwOnwhpD2d12stZuLwzVqhdR0+fBuz5hlOf6op0/5gJ/FVfQIYnFP15Yw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.34) smtp.rcpttodomain=marvell.com smtp.mailfrom=nvidia.com; dmarc=pass (p=quarantine sp=none pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=8TVpBQt+V6f6hp9z2D2qjnhFtrl+F5aIzj80UUbtCE8=; b=tTGvVRvNmG3p3JwsilM/ZfGLD0G7AIR+AEbh4gr/mLUQzMkXrMVD5zPGUYgveMMyFdJIZyCRuPOi+8CaCimN/FgsbUN+Vu5Q99QXTLXp3WZ/Ghn9DNoM1TnjA8POmQJ0G3jaBjaUX8wm5kQ2IK+RJ1WmMfax0JvGFUo3JfLKWgd4nvIc7XRmpldxtbFjvoQyyCSn9zKLCZeJgrxjCKndOhxhA/LCR2mawQNxMORiwdlSKlHVGHpIBS9FqxZen0Y6nFzlB82gj4+3Nk6rhLWxQokgt7KqlZ2H2d8U0eUZaleh43MOoGjQhxckJdvytZEcyRG6kPRQWqPOzZpyFJLHCw== Received: from BN0PR04CA0028.namprd04.prod.outlook.com (2603:10b6:408:ee::33) by CH0PR12MB5139.namprd12.prod.outlook.com (2603:10b6:610:be::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4352.26; Thu, 22 Jul 2021 11:05:03 +0000 Received: from BN8NAM11FT019.eop-nam11.prod.protection.outlook.com (2603:10b6:408:ee:cafe::f8) by BN0PR04CA0028.outlook.office365.com (2603:10b6:408:ee::33) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4352.24 via Frontend Transport; Thu, 22 Jul 2021 11:05:03 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.34) smtp.mailfrom=nvidia.com; marvell.com; dkim=none (message not signed) header.d=none; marvell.com; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.34 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.34; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.34) by BN8NAM11FT019.mail.protection.outlook.com (10.13.176.158) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4352.24 via Frontend Transport; Thu, 22 Jul 2021 11:05:03 +0000 Received: from HQMAIL107.nvidia.com (172.20.187.13) by HQMAIL107.nvidia.com (172.20.187.13) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Thu, 22 Jul 2021 11:05:02 +0000 Received: from vdi.nvidia.com (172.20.187.5) by mail.nvidia.com (172.20.187.13) with Microsoft SMTP Server id 15.0.1497.2 via Frontend Transport; Thu, 22 Jul 2021 11:04:57 +0000 From: Boris Pismenny To: , , , , , , , , , , CC: , , , , , , Boris Pismenny , Or Gerlitz , Yoray Zack Subject: [PATCH v5 net-next 10/36] net/mlx5: Add 128B CQE for NVMEoTCP offload Date: Thu, 22 Jul 2021 14:02:59 +0300 Message-ID: <20210722110325.371-11-borisp@nvidia.com> X-Mailer: git-send-email 2.24.1 In-Reply-To: <20210722110325.371-1-borisp@nvidia.com> References: <20210722110325.371-1-borisp@nvidia.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 3c10297a-77ae-4b6d-2fb9-08d94d008ea5 X-MS-TrafficTypeDiagnostic: CH0PR12MB5139: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:2958; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: E1gTRRqd5+vqWY/zgqd7T4VfOyKJ0KCdxHCpNBFw8TOAZBoR1+PKgiuhF4DyYqFUwdpU5EsJ5aL9q5DMHfL+1uLmNSsqUAZ22gUXkt7cZ05wvtrMMwjoidyM8RJyrGy2r1Dkqypk9ULWMIs4k+raIlvn+QzOfDnYbQ8oYHZ15G8JiVP/DX87Ue+lnuoPG39tylODT8HrmDQX5Lp35FNha7/qTx6LjvwHwFq6Jyc3PwmQAv9ma/zvVsvlJ6tgxrPUnPU8qHsJadh+/H+d/lepkDdQuDzaBiPv2cA4KmszchNrnoRy+Q1RfNaL6SDj1WvXpGLFsTF5VaH+SHfVNnLqmpgiGEFWUENY1/OfG61w2BCsMx593SIQ+zNZg3kmG1fNGzNjlpom4DuFDftkfitgpFY50zRYpE01vdjR2ITfx58wF2mJoTcckz7sIzdq55GxHt++hgOjN6yEkvDnmf0OQRQgXuMLpQxS13X2L9eOdmaxAsTjXrfvqlp0dDHULLiU3fe47G4BXx/VOITzZFMsJNtCaJKIgyHYRYYVLevPd17/aaSoaEweoJ693ql0wll6zssbq++e1yNFdfg5GDZWTlqas29vc+40DjZA1XJFsoETZ0W3Z2ozBTFiscxfAF6R1gDN/sGWMq473/B6nWTqfdRj8dWWpn+zhYTbrjP90rBwkbt6sth7aXg57AaGtUC+sF4/hvFg1hvf55Fwo4zwQEPwqRBS1w/yfBLGWzvocls= X-Forefront-Antispam-Report: CIP:216.228.112.34; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:schybrid03.nvidia.com; CAT:NONE; SFS:(4636009)(346002)(136003)(39860400002)(396003)(376002)(46966006)(36840700001)(478600001)(8676002)(36860700001)(83380400001)(356005)(47076005)(921005)(36906005)(316002)(70206006)(86362001)(7416002)(110136005)(54906003)(5660300002)(70586007)(336012)(4326008)(1076003)(2906002)(186003)(8936002)(2616005)(107886003)(26005)(7696005)(426003)(36756003)(82310400003)(7636003)(82740400003); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Jul 2021 11:05:03.6226 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 3c10297a-77ae-4b6d-2fb9-08d94d008ea5 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.112.34]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BN8NAM11FT019.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH0PR12MB5139 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Ben Ben-ishay Add the NVMEoTCP offload definition and access functions for 128B CQEs. Signed-off-by: Ben Ben-ishay Signed-off-by: Boris Pismenny Signed-off-by: Or Gerlitz Signed-off-by: Yoray Zack --- include/linux/mlx5/device.h | 36 +++++++++++++++++++++++++++++++++++- 1 file changed, 35 insertions(+), 1 deletion(-) diff --git a/include/linux/mlx5/device.h b/include/linux/mlx5/device.h index a42e47f91327..412d0982ee46 100644 --- a/include/linux/mlx5/device.h +++ b/include/linux/mlx5/device.h @@ -794,7 +794,7 @@ struct mlx5_err_cqe { struct mlx5_cqe64 { u8 tls_outer_l3_tunneled; - u8 rsvd0; + u8 nvmetcp; __be16 wqe_id; u8 lro_tcppsh_abort_dupack; u8 lro_min_ttl; @@ -827,6 +827,19 @@ struct mlx5_cqe64 { u8 op_own; }; +struct mlx5e_cqe128 { + __be16 cclen; + __be16 hlen; + union { + __be32 resync_tcp_sn; + __be32 ccoff; + }; + __be16 ccid; + __be16 rsvd8; + u8 rsvd12[52]; + struct mlx5_cqe64 cqe64; +}; + struct mlx5_mini_cqe8 { union { __be32 rx_hash_result; @@ -857,6 +870,27 @@ enum { #define MLX5_MINI_CQE_ARRAY_SIZE 8 +static inline bool cqe_is_nvmeotcp_resync(struct mlx5_cqe64 *cqe) +{ + return ((cqe->nvmetcp >> 6) & 0x1); +} + +static inline bool cqe_is_nvmeotcp_crcvalid(struct mlx5_cqe64 *cqe) +{ + return ((cqe->nvmetcp >> 5) & 0x1); +} + +static inline bool cqe_is_nvmeotcp_zc(struct mlx5_cqe64 *cqe) +{ + return ((cqe->nvmetcp >> 4) & 0x1); +} + +/* check if cqe is zc or crc or resync */ +static inline bool cqe_is_nvmeotcp(struct mlx5_cqe64 *cqe) +{ + return ((cqe->nvmetcp >> 4) & 0x7); +} + static inline u8 mlx5_get_cqe_format(struct mlx5_cqe64 *cqe) { return (cqe->op_own >> 2) & 0x3; From patchwork Thu Jul 22 11:03:00 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Boris Pismenny X-Patchwork-Id: 484321 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-20.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER, INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id DC172C6377D for ; Thu, 22 Jul 2021 11:05:23 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id C398760725 for ; Thu, 22 Jul 2021 11:05:23 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231759AbhGVKYr (ORCPT ); Thu, 22 Jul 2021 06:24:47 -0400 Received: from mail-dm6nam12on2058.outbound.protection.outlook.com ([40.107.243.58]:31968 "EHLO NAM12-DM6-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S231718AbhGVKYe (ORCPT ); Thu, 22 Jul 2021 06:24:34 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Wa9Kjch7NcPik1grpx8A9lv6SCR3gxgxWuaSB4JdsU4R0Z1MQrE0/MPV1dF6vXEjwmphxyE1IMt77yLNb1H84A7ewfOtv3jg7zJvf8Wo1dEvQkT1unHfme2FK3iTh0IRQw3Lcqy6PtNRwpdg4Osn/h6QpIhANcWuA5baOQ2b2AfG/g5Y5Grv8g5scde2Bm1ieDUUsZJKv+gUPhfIjNPA8AE1JbdwS5F2HQYLPNDqI9yiLBqtey3M/8nkA8QquTMRy7K2WnSWfDNjNjqa8/QPnAW/tvzKssA3gTxoTqo9fmXZNJnQYbUUAVN4VtSIsIv+WAo3J7tOC0NVjvETnasOrg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=w9dSDgtM8uufd6jUhzV5bb5tlCJSXGPodLbDpo4FHWw=; b=gBzMshFWlEsZEyzu9HW9RLcOeRcCggmNWm4+ZoBKB5aIN1ih9eFjr70gOkb/WEvwETp7i5b8XP+dN9+0RXbhWoAffVjyUMfqNAnV8CYf2qq8cBFOpmo9opJLngJY6HJtMzAiPS0TIMFDXMjo9xQeAz/E5O5hzOiHJOr11U5RoU4iRQwJlWs0JuWvrKdJTJPseKhf8kJRMS8qq7mdcn6+bJMkrWjGcA09k9pFp8HDfvTbDsFby2zNrqmiEuM3dzqNiB7jNCTSTZGtFy0FPOPp8Y+mebU4FWDFBEpRK5TiLthSm0x/HgvSeHfpXSrc35xvQapZB5OBqo5mapbw0Y591A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.36) smtp.rcpttodomain=marvell.com smtp.mailfrom=nvidia.com; dmarc=pass (p=quarantine sp=none pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=w9dSDgtM8uufd6jUhzV5bb5tlCJSXGPodLbDpo4FHWw=; b=HZoI/KtyLoK5KFG45FX/fIPTCG/4dOy8y7eZs9qF4l9dhfP3l/+54ALUKTI2+URe6VA+hXxf+gb9jgBkbbxAaLTwpGquJhe64lyVa5m2ujT8eL8ODAmv9ii1vs2WfscvuhV9bUai5/izH0wD7vN/fSk+7t16UXVTOy0qUksepOqqnvhbxeDxhWrRCMIXM5lqZEKu+lDzzYM8kdoZXA7hzRQtDTUDaITOcoUK2Qipl1Wy/A0T2Jpx4cArSSFCyD1EZD3YsJQbAzdSxSWsp07yESDG3pJcia/JEJ4N+sM2OCMeEoVx0HFal82ZHHSdWQJBo5HPRaFhAAIb+ZAhBVGKsw== Received: from BN9PR03CA0666.namprd03.prod.outlook.com (2603:10b6:408:10e::11) by DM6PR12MB4651.namprd12.prod.outlook.com (2603:10b6:5:1f6::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4331.23; Thu, 22 Jul 2021 11:05:08 +0000 Received: from BN8NAM11FT052.eop-nam11.prod.protection.outlook.com (2603:10b6:408:10e:cafe::b4) by BN9PR03CA0666.outlook.office365.com (2603:10b6:408:10e::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4331.22 via Frontend Transport; Thu, 22 Jul 2021 11:05:08 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.36) smtp.mailfrom=nvidia.com; marvell.com; dkim=none (message not signed) header.d=none; marvell.com; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.36 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.36; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.36) by BN8NAM11FT052.mail.protection.outlook.com (10.13.177.210) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4352.24 via Frontend Transport; Thu, 22 Jul 2021 11:05:07 +0000 Received: from HQMAIL107.nvidia.com (172.20.187.13) by HQMAIL101.nvidia.com (172.20.187.10) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Thu, 22 Jul 2021 11:05:07 +0000 Received: from vdi.nvidia.com (172.20.187.5) by mail.nvidia.com (172.20.187.13) with Microsoft SMTP Server id 15.0.1497.2 via Frontend Transport; Thu, 22 Jul 2021 11:05:02 +0000 From: Boris Pismenny To: , , , , , , , , , , CC: , , , , , , Boris Pismenny , Ben Ben-Ishay , Or Gerlitz , Yoray Zack Subject: [PATCH v5 net-next 11/36] net/mlx5e: TCP flow steering for nvme-tcp Date: Thu, 22 Jul 2021 14:03:00 +0300 Message-ID: <20210722110325.371-12-borisp@nvidia.com> X-Mailer: git-send-email 2.24.1 In-Reply-To: <20210722110325.371-1-borisp@nvidia.com> References: <20210722110325.371-1-borisp@nvidia.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: f8217087-c568-4a94-d30d-08d94d009127 X-MS-TrafficTypeDiagnostic: DM6PR12MB4651: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:632; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 8ChfRYvNHS3TDLOgzDCiXAggr9Si3p28gA+eVBxOvEqz2q4RYkHwUEqRGJ1UBt+l1oWWE7yuxbiEvIOte9jprPLdnMW9KRmupYVp5SeOXf1OWGgkjLjpkBtZMykKlZkiYiJf0UFvEc5Jnfd1r78JAy4qj3pOqlfid//sbpwImKoEf5KxZ7j+7moFBr0sghkzonII6gLFsxW4QTCUbGukc8+rJ9tTwcNbOsmuJRMlqRoptpDYogrvUg9RwpgWwW8IJGVwU0EtMikD4785/2gAQ+V/1yXPOJanu9P9mbQcCS3fPNSMBEy0l7NqOkX9WpizI1eMpuqi8/J+4p4nn/xDG/ojDAV79xoleUaroBZ9You6aYpVTDQLLtMHLCVF7ygECYg3Og1s17gcJ7exMW4/a7L7elYdGLpCgQaau+TX0vSxPO+kRbmQBUlGFClrJyP4ORqk+C6G5udGTOul0D8KzdjJowxnEsvITlOF+Ab+AU8ejynzsxSqM0Zk7KQ6UkSbQ0Tr0zn30n/K7ZGeMJSUTKUfCp3CqinxoxoP0ogmxKDQBdq6/uwqXBw8bHioMs/i8CQI7fk4Ek4+INa7AsL+TF3+y6Z2TYXgO7AhYCgeVuQiKDLjigo2tB6wYOEJ2FGNV+lPkX3Mr6S2bwqMEmjzqXpDbK0yuXsT/my7Ao4fq17rMV+2+z89/bDhrAF9spMC/mJHcyh28txjY6oPSrSNPVXkRm4pvBzaKoBky4Wjw4E= X-Forefront-Antispam-Report: CIP:216.228.112.36; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:schybrid05.nvidia.com; CAT:NONE; SFS:(4636009)(46966006)(36840700001)(5660300002)(107886003)(82310400003)(83380400001)(2906002)(356005)(426003)(7636003)(2616005)(8676002)(186003)(47076005)(86362001)(54906003)(70206006)(7696005)(1076003)(4326008)(336012)(36756003)(26005)(110136005)(316002)(508600001)(70586007)(36860700001)(921005)(7416002)(8936002)(36906005); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Jul 2021 11:05:07.8661 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: f8217087-c568-4a94-d30d-08d94d009127 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.112.36]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BN8NAM11FT052.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR12MB4651 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Boris Pismenny Both nvme-tcp and tls require tcp flow steering. Compile it for both of them. Additionally, use reference counting to allocate/free TCP flow steering. Signed-off-by: Boris Pismenny Signed-off-by: Ben Ben-Ishay Signed-off-by: Or Gerlitz Signed-off-by: Yoray Zack --- drivers/net/ethernet/mellanox/mlx5/core/en/fs.h | 4 ++-- .../net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.c | 10 ++++++++++ .../net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.h | 2 +- 3 files changed, 13 insertions(+), 3 deletions(-) diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/fs.h b/drivers/net/ethernet/mellanox/mlx5/core/en/fs.h index 1d5ce07b83f4..8690919f2cde 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en/fs.h +++ b/drivers/net/ethernet/mellanox/mlx5/core/en/fs.h @@ -132,7 +132,7 @@ enum { MLX5E_INNER_TTC_FT_LEVEL, MLX5E_FS_TT_UDP_FT_LEVEL = MLX5E_INNER_TTC_FT_LEVEL + 1, MLX5E_FS_TT_ANY_FT_LEVEL = MLX5E_INNER_TTC_FT_LEVEL + 1, -#ifdef CONFIG_MLX5_EN_TLS +#if defined(CONFIG_MLX5_EN_TLS) || defined(CONFIG_MLX5_EN_NVMEOTCP) MLX5E_ACCEL_FS_TCP_FT_LEVEL = MLX5E_INNER_TTC_FT_LEVEL + 1, #endif #ifdef CONFIG_MLX5_EN_ARFS @@ -231,7 +231,7 @@ struct mlx5e_flow_steering { #ifdef CONFIG_MLX5_EN_ARFS struct mlx5e_arfs_tables *arfs; #endif -#ifdef CONFIG_MLX5_EN_TLS +#if defined(CONFIG_MLX5_EN_TLS) || defined(CONFIG_MLX5_EN_NVMEOTCP) struct mlx5e_accel_fs_tcp *accel_tcp; #endif struct mlx5e_fs_udp *udp; diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.c b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.c index e51f60b55daa..21341a92f355 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.c @@ -14,6 +14,7 @@ enum accel_fs_tcp_type { struct mlx5e_accel_fs_tcp { struct mlx5e_flow_table tables[ACCEL_FS_TCP_NUM_TYPES]; struct mlx5_flow_handle *default_rules[ACCEL_FS_TCP_NUM_TYPES]; + refcount_t ref_count; }; static enum mlx5e_traffic_types fs_accel2tt(enum accel_fs_tcp_type i) @@ -337,6 +338,7 @@ static int accel_fs_tcp_enable(struct mlx5e_priv *priv) return err; } } + refcount_set(&priv->fs.accel_tcp->ref_count, 1); return 0; } @@ -360,6 +362,9 @@ void mlx5e_accel_fs_tcp_destroy(struct mlx5e_priv *priv) if (!priv->fs.accel_tcp) return; + if (!refcount_dec_and_test(&priv->fs.accel_tcp->ref_count)) + return; + accel_fs_tcp_disable(priv); for (i = 0; i < ACCEL_FS_TCP_NUM_TYPES; i++) @@ -376,6 +381,11 @@ int mlx5e_accel_fs_tcp_create(struct mlx5e_priv *priv) if (!MLX5_CAP_FLOWTABLE_NIC_RX(priv->mdev, ft_field_support.outer_ip_version)) return -EOPNOTSUPP; + if (priv->fs.accel_tcp) { + refcount_inc(&priv->fs.accel_tcp->ref_count); + return 0; + } + priv->fs.accel_tcp = kzalloc(sizeof(*priv->fs.accel_tcp), GFP_KERNEL); if (!priv->fs.accel_tcp) return -ENOMEM; diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.h b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.h index 589235824543..8aff9298183c 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.h +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.h @@ -6,7 +6,7 @@ #include "en.h" -#ifdef CONFIG_MLX5_EN_TLS +#if defined(CONFIG_MLX5_EN_TLS) || defined(CONFIG_MLX5_EN_NVMEOTCP) int mlx5e_accel_fs_tcp_create(struct mlx5e_priv *priv); void mlx5e_accel_fs_tcp_destroy(struct mlx5e_priv *priv); struct mlx5_flow_handle *mlx5e_accel_fs_add_sk(struct mlx5e_priv *priv, From patchwork Thu Jul 22 11:03:01 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Boris Pismenny X-Patchwork-Id: 484320 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-20.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER, INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A8723C63793 for ; Thu, 22 Jul 2021 11:05:26 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 8D51C60725 for ; Thu, 22 Jul 2021 11:05:26 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231772AbhGVKYu (ORCPT ); Thu, 22 Jul 2021 06:24:50 -0400 Received: from mail-mw2nam12on2068.outbound.protection.outlook.com ([40.107.244.68]:54241 "EHLO NAM12-MW2-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S231751AbhGVKYj (ORCPT ); Thu, 22 Jul 2021 06:24:39 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=nkcTYaOJ6Hqs4yeh8Y2rmwYU1zuAArHWtal8fIERAgqMXgRGWJdzd1zHwHZOrw4uJx/DTP2CWWEd0DE12BgatykO5rfWBc3tml9eT1FIOWvY5Tdb0eKm3LeOsgsvdUrPFtHW2dg8TAV7y8m6qSWNFzLchaOyXM5vRBgYPPfo0cpQJcpw+o4E+timhZXFaxyg9r6HKGcV+UrMwoyydxU4bpN40d+4Uk5JwK8LIyxgFguX6YYfb2J5RYF8uJUh0BbsK1JJugwzkfACWcBACvR4pDLSy+zosgpEYhZ/9Zo6rKUAZgOnCU0w01PvZgsRVcizHJRdgAY+jXyxc0cJ5m5mIA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=JPpZMFDexWhzDg1HOgN0yPn8umTF5Qbp4zxP4N/zjE4=; b=eKAJGtMb4cqMox7OkSKejgsBX5HCefUuCDTHZVTvKWpAW6EnM5iaVG5xzDnsPp5eXplaXgbyGtTjf17v+J+9/0+Hw3XgVCjtsj+EKhgcKAzIhaPKWk57fzvGnVnd4mOJIbg3FCfWLZpDmz1+oEBNF8NMc81NXGqg5oUyAcFPhvvj+LEmmGg3OZy/e9/DBh90f6MRETBuegftYqIhLJ/KiYtr0eQf1/ESPlTi35fheWT8i+tETHZWwEwFRjUl1NzTK1zZbb8MI9NcVQnx7JBhrpNe3FtORl2+j3bcV8a7HqQBgE/Xlr8UGbnlIapHLaYnkaRkquNv1VH67yq9qYorig== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.34) smtp.rcpttodomain=marvell.com smtp.mailfrom=nvidia.com; dmarc=pass (p=quarantine sp=none pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=JPpZMFDexWhzDg1HOgN0yPn8umTF5Qbp4zxP4N/zjE4=; b=EbJrdG4DD75Ks0krhIT3Z0rHrfp6zgyf9SU/BMs6mXVLrgnogQtGK/Tu1AD2Bxf35ENCQO7jbxX+2ZzT6f8GiOy9SI6syWnFRsEaGS77v+XY1aTtUdlBXj/q/iZAOnWI4VDVHLBFM7Wn7elYyrYn9VKrOoIt6IoQB+EOSpaF4JBV2ykOEZv3lghlJ2IQE7yJmXvqaza7PuK32ROsD+4U7GKvFK5KmsNt5+uk0meoZ5n0GsSS2ZaF8o7xu+xfVz34mUpssAFXVxHU9Mytb2HQjbUI3suSfWzBjKKFSbd2vdPXXymH7v5rMkWSDQV+SCUglQER56/OcAYRtUWkdtCZRw== Received: from BN0PR04CA0005.namprd04.prod.outlook.com (2603:10b6:408:ee::10) by BL1PR12MB5173.namprd12.prod.outlook.com (2603:10b6:208:308::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4331.22; Thu, 22 Jul 2021 11:05:12 +0000 Received: from BN8NAM11FT019.eop-nam11.prod.protection.outlook.com (2603:10b6:408:ee:cafe::5c) by BN0PR04CA0005.outlook.office365.com (2603:10b6:408:ee::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4331.23 via Frontend Transport; Thu, 22 Jul 2021 11:05:12 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.34) smtp.mailfrom=nvidia.com; marvell.com; dkim=none (message not signed) header.d=none; marvell.com; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.34 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.34; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.34) by BN8NAM11FT019.mail.protection.outlook.com (10.13.176.158) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4352.24 via Frontend Transport; Thu, 22 Jul 2021 11:05:12 +0000 Received: from HQMAIL107.nvidia.com (172.20.187.13) by HQMAIL107.nvidia.com (172.20.187.13) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Thu, 22 Jul 2021 11:05:11 +0000 Received: from vdi.nvidia.com (172.20.187.5) by mail.nvidia.com (172.20.187.13) with Microsoft SMTP Server id 15.0.1497.2 via Frontend Transport; Thu, 22 Jul 2021 11:05:07 +0000 From: Boris Pismenny To: , , , , , , , , , , CC: , , , , , , Boris Pismenny , Or Gerlitz , Yoray Zack Subject: [PATCH v5 net-next 12/36] net/mlx5e: NVMEoTCP offload initialization Date: Thu, 22 Jul 2021 14:03:01 +0300 Message-ID: <20210722110325.371-13-borisp@nvidia.com> X-Mailer: git-send-email 2.24.1 In-Reply-To: <20210722110325.371-1-borisp@nvidia.com> References: <20210722110325.371-1-borisp@nvidia.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: a9cf560c-4007-4d40-d0a3-08d94d0093f6 X-MS-TrafficTypeDiagnostic: BL1PR12MB5173: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:268; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 3lWZqYRSmYl/3OJtTWmsBsod0gZJsEgji0jUhq0dQwwv8zzouFU5DZuB+VUkvr/0zrRmNcWHv4NkfGgMX0khHSoCuP5uotPjT6U5UyGbhg8+4IuWe5vE0uNec6Vakmd4YJ8XY58MaTojFhXxGUVJrSeA7m1U4Eq9v7rZXzGoPR6lY4dZnrDuJKWPbqk0GLCjyc81S1H6Xj3nMDfLjHjaihOewb81TaARHdwQlistFkyT53OLj0IUADJVKdjFhjxwFlEGf2tH+299xYwHCX00Pl7w3CVam0X1TCDPwXMUc5gyzp4JdInbSbkhA47RWKvxlPElM+jdFpM86qlpQPVCoLzS//KGpsFbrg6xvTnlOHo+8ME/RiTfiK3V+tTrl2oSSyIh2ag86khj1rjj0uhZ1zuKvnOsyb3wb20qLsKHNqGRcoHOtqhTXkCeItKs4EncnveMsdvGgw2o6hpD0sx6C5uF3Bchj3+cOYjp/x6227GBTFLWsPLP6kT+f1G670395z66JZP8lwRRZPzSX2gWvOk1a3xrbdi9IitW+OrLon6V40IL+F1skJOWTQu/D/lPOoEab4ScgVUKiO0XfKjLpzwGPU1BLnt86H964/qe5uxcxB1bZUkTiQD4sxv6W920HlCORU6AyLfSr4RPURLmKxxylk95ZcyjJmbxvwaBXQ1JCZvhifbc4sy4LSOM/rfz9uqRRJy+ucCwv5KQThBTCKWKJTa3eor9WoUMwrpxS8A= X-Forefront-Antispam-Report: CIP:216.228.112.34; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:schybrid03.nvidia.com; CAT:NONE; SFS:(4636009)(376002)(346002)(136003)(396003)(39860400002)(36840700001)(46966006)(356005)(7636003)(83380400001)(82310400003)(70586007)(86362001)(70206006)(7416002)(316002)(6666004)(426003)(336012)(82740400003)(4326008)(2906002)(107886003)(8676002)(36756003)(54906003)(110136005)(36906005)(1076003)(30864003)(921005)(8936002)(47076005)(7696005)(186003)(2616005)(478600001)(36860700001)(26005)(5660300002); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Jul 2021 11:05:12.5771 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: a9cf560c-4007-4d40-d0a3-08d94d0093f6 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.112.34]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BN8NAM11FT019.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL1PR12MB5173 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Ben Ben-Ishay This commit introduce the initialization blocks for NVMEoTCP offload: - Use 128B CQEs when NVME-TCP offload is enabled. - Use a dedicated icosq for NVME-TCP work. This list of SQ is unique in the sense that it is driven directly by the NVME-TCP layer to submit and invalidate ddp requests. - Query nvmeotcp capabilities Signed-off-by: Boris Pismenny Signed-off-by: Ben Ben-Ishay Signed-off-by: Or Gerlitz Signed-off-by: Yoray Zack --- .../net/ethernet/mellanox/mlx5/core/Kconfig | 10 + .../net/ethernet/mellanox/mlx5/core/Makefile | 2 + drivers/net/ethernet/mellanox/mlx5/core/en.h | 11 + .../ethernet/mellanox/mlx5/core/en/params.c | 11 +- .../ethernet/mellanox/mlx5/core/en/params.h | 3 + .../mellanox/mlx5/core/en_accel/en_accel.h | 9 +- .../mellanox/mlx5/core/en_accel/nvmeotcp.c | 196 ++++++++++++++++++ .../mellanox/mlx5/core/en_accel/nvmeotcp.h | 117 +++++++++++ .../net/ethernet/mellanox/mlx5/core/en_main.c | 26 +++ .../net/ethernet/mellanox/mlx5/core/en_txrx.c | 17 ++ drivers/net/ethernet/mellanox/mlx5/core/fw.c | 6 + 11 files changed, 405 insertions(+), 3 deletions(-) create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.h diff --git a/drivers/net/ethernet/mellanox/mlx5/core/Kconfig b/drivers/net/ethernet/mellanox/mlx5/core/Kconfig index e1a5a79e27c7..e6079ff2e917 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/Kconfig +++ b/drivers/net/ethernet/mellanox/mlx5/core/Kconfig @@ -233,3 +233,13 @@ config MLX5_SF_MANAGER port is managed through devlink. A subfunction supports RDMA, netdevice and vdpa device. It is similar to a SRIOV VF but it doesn't require SRIOV support. + +config MLX5_EN_NVMEOTCP + bool "NVMEoTCP accelaration" + depends on MLX5_CORE_EN + depends on ULP_DDP=y + default n + help + Build support for NVMEoTCP accelaration in the NIC. + Note: Support for hardware with this capability needs to be selected + for this option to become available. diff --git a/drivers/net/ethernet/mellanox/mlx5/core/Makefile b/drivers/net/ethernet/mellanox/mlx5/core/Makefile index b5072a3a2585..0ae9e5e38ec7 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/Makefile +++ b/drivers/net/ethernet/mellanox/mlx5/core/Makefile @@ -101,3 +101,5 @@ mlx5_core-$(CONFIG_MLX5_SF) += sf/vhca_event.o sf/dev/dev.o sf/dev/driver.o # SF manager # mlx5_core-$(CONFIG_MLX5_SF_MANAGER) += sf/cmd.o sf/hw_table.o sf/devlink.o + +mlx5_core-$(CONFIG_MLX5_EN_NVMEOTCP) += en_accel/fs_tcp.o en_accel/nvmeotcp.o diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en.h b/drivers/net/ethernet/mellanox/mlx5/core/en.h index b1b51bbba054..1233ebcf311b 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en.h +++ b/drivers/net/ethernet/mellanox/mlx5/core/en.h @@ -270,6 +270,10 @@ struct mlx5e_params { unsigned int sw_mtu; int hard_mtu; bool ptp_rx; +#ifdef CONFIG_MLX5_EN_NVMEOTCP + bool nvmeotcp; + bool crc_rx_offload; +#endif }; enum { @@ -678,6 +682,10 @@ struct mlx5e_channel { struct mlx5e_txqsq sq[MLX5E_MAX_NUM_TC]; struct mlx5e_icosq icosq; /* internal control operations */ struct mlx5e_txqsq __rcu * __rcu *qos_sqs; +#ifdef CONFIG_MLX5_EN_NVMEOTCP + struct list_head list_nvmeotcpsq; /* nvmeotcp umrs */ + spinlock_t nvmeotcp_icosq_lock; +#endif bool xdp; struct napi_struct napi; struct device *pdev; @@ -886,6 +894,9 @@ struct mlx5e_priv { #endif #ifdef CONFIG_MLX5_EN_TLS struct mlx5e_tls *tls; +#endif +#ifdef CONFIG_MLX5_EN_NVMEOTCP + struct mlx5e_nvmeotcp *nvmeotcp; #endif struct devlink_health_reporter *tx_reporter; struct devlink_health_reporter *rx_reporter; diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/params.c b/drivers/net/ethernet/mellanox/mlx5/core/en/params.c index 150c8e82c738..a84508425e47 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en/params.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en/params.c @@ -437,7 +437,8 @@ static void mlx5e_build_common_cq_param(struct mlx5_core_dev *mdev, void *cqc = param->cqc; MLX5_SET(cqc, cqc, uar_page, mdev->priv.uar->index); - if (MLX5_CAP_GEN(mdev, cqe_128_always) && cache_line_size() >= 128) + if (MLX5_CAP_GEN(mdev, cqe_128_always) && + (cache_line_size() >= 128 || param->force_cqe128)) MLX5_SET(cqc, cqc, cqe_sz, CQE_STRIDE_128_PAD); } @@ -450,6 +451,12 @@ static void mlx5e_build_rx_cq_param(struct mlx5_core_dev *mdev, void *cqc = param->cqc; u8 log_cq_size; +#ifdef CONFIG_MLX5_EN_NVMEOTCP + /* nvme-tcp offload mandates 128 byte cqes */ + param->force_cqe128 |= (params->nvmeotcp|| params->crc_rx_offload); +#endif + + switch (params->rq_wq_type) { case MLX5_WQ_TYPE_LINKED_LIST_STRIDING_RQ: log_cq_size = mlx5e_mpwqe_get_log_rq_size(params, xsk) + @@ -620,7 +627,7 @@ static u8 mlx5e_build_async_icosq_log_wq_sz(struct mlx5_core_dev *mdev) return MLX5E_PARAMS_MINIMUM_LOG_SQ_SIZE; } -static void mlx5e_build_icosq_param(struct mlx5_core_dev *mdev, +void mlx5e_build_icosq_param(struct mlx5_core_dev *mdev, u8 log_wq_size, struct mlx5e_sq_param *param) { diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/params.h b/drivers/net/ethernet/mellanox/mlx5/core/en/params.h index e9593f5f0661..4f232ba726ec 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en/params.h +++ b/drivers/net/ethernet/mellanox/mlx5/core/en/params.h @@ -16,6 +16,7 @@ struct mlx5e_cq_param { struct mlx5_wq_param wq; u16 eq_ix; u8 cq_period_mode; + bool force_cqe128; }; struct mlx5e_rq_param { @@ -147,6 +148,8 @@ int mlx5e_build_channel_param(struct mlx5_core_dev *mdev, struct mlx5e_params *params, u16 q_counter, struct mlx5e_channel_param *cparam); +void mlx5e_build_icosq_param(struct mlx5_core_dev *mdev, + u8 log_wq_size,struct mlx5e_sq_param *param); u16 mlx5e_calc_sq_stop_room(struct mlx5_core_dev *mdev, struct mlx5e_params *params); int mlx5e_validate_params(struct mlx5_core_dev *mdev, struct mlx5e_params *params); diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/en_accel.h b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/en_accel.h index d964665eaa63..b9404366e6e8 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/en_accel.h +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/en_accel.h @@ -39,6 +39,7 @@ #include "en_accel/ipsec_rxtx.h" #include "en_accel/tls.h" #include "en_accel/tls_rxtx.h" +#include "en_accel/nvmeotcp.h" #include "en.h" #include "en/txrx.h" @@ -195,11 +196,17 @@ static inline void mlx5e_accel_tx_finish(struct mlx5e_txqsq *sq, static inline int mlx5e_accel_init_rx(struct mlx5e_priv *priv) { - return mlx5e_ktls_init_rx(priv); + int tls, nvmeotcp; + + tls = mlx5e_ktls_init_rx(priv); + nvmeotcp = mlx5e_nvmeotcp_init_rx(priv); + + return tls && nvmeotcp; } static inline void mlx5e_accel_cleanup_rx(struct mlx5e_priv *priv) { + mlx5e_nvmeotcp_cleanup_rx(priv); mlx5e_ktls_cleanup_rx(priv); } #endif /* __MLX5E_EN_ACCEL_H__ */ diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c new file mode 100644 index 000000000000..04e88042b243 --- /dev/null +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c @@ -0,0 +1,196 @@ +// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB +/* Copyright (c) 2021 Mellanox Technologies. */ + +#include +#include +#include "en_accel/nvmeotcp.h" +#include "en_accel/fs_tcp.h" +#include "en/txrx.h" + +#define MAX_NVMEOTCP_QUEUES (512) +#define MIN_NVMEOTCP_QUEUES (1) + +static const struct rhashtable_params rhash_queues = { + .key_len = sizeof(int), + .key_offset = offsetof(struct mlx5e_nvmeotcp_queue, id), + .head_offset = offsetof(struct mlx5e_nvmeotcp_queue, hash), + .automatic_shrinking = true, + .min_size = 1, + .max_size = MAX_NVMEOTCP_QUEUES, +}; + +static int +mlx5e_nvmeotcp_offload_limits(struct net_device *netdev, + struct ulp_ddp_limits *limits) +{ + return 0; +} + +static int +mlx5e_nvmeotcp_queue_init(struct net_device *netdev, + struct sock *sk, + struct ulp_ddp_config *tconfig) +{ + return 0; +} + +static void +mlx5e_nvmeotcp_queue_teardown(struct net_device *netdev, + struct sock *sk) +{ +} + +static int +mlx5e_nvmeotcp_ddp_setup(struct net_device *netdev, + struct sock *sk, + struct ulp_ddp_io *ddp) +{ + return 0; +} + +static int +mlx5e_nvmeotcp_ddp_teardown(struct net_device *netdev, + struct sock *sk, + struct ulp_ddp_io *ddp, + void *ddp_ctx) +{ + return 0; +} + +static void +mlx5e_nvmeotcp_dev_resync(struct net_device *netdev, + struct sock *sk, u32 seq) +{ +} + +static const struct ulp_ddp_dev_ops mlx5e_nvmeotcp_ops = { + .ulp_ddp_limits = mlx5e_nvmeotcp_offload_limits, + .ulp_ddp_sk_add = mlx5e_nvmeotcp_queue_init, + .ulp_ddp_sk_del = mlx5e_nvmeotcp_queue_teardown, + .ulp_ddp_setup = mlx5e_nvmeotcp_ddp_setup, + .ulp_ddp_teardown = mlx5e_nvmeotcp_ddp_teardown, + .ulp_ddp_resync = mlx5e_nvmeotcp_dev_resync, +}; + +int set_feature_nvme_tcp(struct net_device *netdev, bool enable) +{ + struct mlx5e_priv *priv = netdev_priv(netdev); + struct mlx5e_params *params = &priv->channels.params; + int err = 0; + + mutex_lock(&priv->state_lock); + if (enable) + err = mlx5e_accel_fs_tcp_create(priv); + else + mlx5e_accel_fs_tcp_destroy(priv); + mutex_unlock(&priv->state_lock); + if (err) + return err; + + params->nvmeotcp = enable; + priv->nvmeotcp->enable = enable; + err = mlx5e_safe_reopen_channels(priv); + return err; +} + +int set_feature_nvme_tcp_crc(struct net_device *netdev, bool enable) +{ + struct mlx5e_priv *priv = netdev_priv(netdev); + struct mlx5e_params *params = &priv->channels.params; + int err = 0; + + mutex_lock(&priv->state_lock); + if (enable) + err = mlx5e_accel_fs_tcp_create(priv); + else + mlx5e_accel_fs_tcp_destroy(priv); + mutex_unlock(&priv->state_lock); + + params->crc_rx_offload = enable; + priv->nvmeotcp->crc_rx_enable = enable; + err = mlx5e_safe_reopen_channels(priv); + if (err) + netdev_err(priv->netdev, + "%s failed to reopen channels, err(%d).\n", + __func__, err); + + return err; +} + +void mlx5e_nvmeotcp_build_netdev(struct mlx5e_priv *priv) +{ + struct net_device *netdev = priv->netdev; + + if (!MLX5_CAP_GEN(priv->mdev, nvmeotcp)) + return; + + if (MLX5_CAP_DEV_NVMEOTCP(priv->mdev, zerocopy)) { + netdev->features |= NETIF_F_HW_ULP_DDP; + netdev->hw_features |= NETIF_F_HW_ULP_DDP; + } + + if (MLX5_CAP_DEV_NVMEOTCP(priv->mdev, crc_rx)) { + netdev->features |= NETIF_F_HW_ULP_DDP; + netdev->hw_features |= NETIF_F_HW_ULP_DDP; + } + + netdev->ulp_ddp_ops = &mlx5e_nvmeotcp_ops; +} + +int mlx5e_nvmeotcp_init_rx(struct mlx5e_priv *priv) +{ + int ret = 0; + + if (priv->netdev->features & NETIF_F_HW_ULP_DDP) { + ret = mlx5e_accel_fs_tcp_create(priv); + if (ret) + return ret; + } + + return ret; +} + +void mlx5e_nvmeotcp_cleanup_rx(struct mlx5e_priv *priv) +{ + if (priv->netdev->features & NETIF_F_HW_ULP_DDP) + mlx5e_accel_fs_tcp_destroy(priv); +} + +int mlx5e_nvmeotcp_init(struct mlx5e_priv *priv) +{ + struct mlx5e_nvmeotcp *nvmeotcp = kzalloc(sizeof(*nvmeotcp), GFP_KERNEL); + int ret = 0; + + if (!nvmeotcp) + return -ENOMEM; + + ida_init(&nvmeotcp->queue_ids); + ret = rhashtable_init(&nvmeotcp->queue_hash, &rhash_queues); + if (ret) + goto err_ida; + + priv->nvmeotcp = nvmeotcp; + priv->nvmeotcp->enable = true; + priv->channels.params.nvmeotcp = nvmeotcp; + priv->channels.params.nvmeotcp = true; + goto out; + +err_ida: + ida_destroy(&nvmeotcp->queue_ids); + kfree(nvmeotcp); +out: + return ret; +} + +void mlx5e_nvmeotcp_cleanup(struct mlx5e_priv *priv) +{ + struct mlx5e_nvmeotcp *nvmeotcp = priv->nvmeotcp; + + if (!nvmeotcp) + return; + + rhashtable_destroy(&nvmeotcp->queue_hash); + ida_destroy(&nvmeotcp->queue_ids); + kfree(nvmeotcp); + priv->nvmeotcp = NULL; +} diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.h b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.h new file mode 100644 index 000000000000..b4a27a03578e --- /dev/null +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.h @@ -0,0 +1,117 @@ +/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */ +/* Copyright (c) 2021 Mellanox Technologies. */ +#ifndef __MLX5E_NVMEOTCP_H__ +#define __MLX5E_NVMEOTCP_H__ + +#ifdef CONFIG_MLX5_EN_NVMEOTCP + +#include "net/ulp_ddp.h" +#include "en.h" +#include "en/params.h" + +struct nvmeotcp_queue_entry { + struct mlx5e_nvmeotcp_queue *queue; + u32 sgl_length; + struct mlx5_core_mkey klm_mkey; + struct scatterlist *sgl; + u32 ccid_gen; + u64 size; + + /* for the ddp invalidate done callback */ + void *ddp_ctx; + struct ulp_ddp_io *ddp; +}; + +struct mlx5e_nvmeotcp_sq { + struct list_head list; + struct mlx5e_icosq icosq; +}; + +/** + * struct mlx5e_nvmeotcp_queue - MLX5 metadata for NVMEoTCP queue + * @fh: Flow handle representing the 5-tuple steering for this flow + * @tirn: Destination TIR number created for NVMEoTCP offload + * @id: Flow tag ID used to identify this queue + * @size: NVMEoTCP queue depth + * @sq: Send queue used for sending control messages + * @nvmeotcp_icosq_lock: Spin lock for icosq + * @ccid_table: Table holding metadata for each CC + * @tag_buf_table_id: Tag buffer table for CCIDs + * @hash: Hash table of queues mapped by @id + * @ref_count: Reference count for this structure + * @ccoff: Offset within the current CC + * @pda: Padding alignment + * @ccid_gen: Generation ID for the CCID, used to avoid conflicts in DDP + * @max_klms_per_wqe: Number of KLMs per DDP operation + * @channel_ix: Channel IX for this nvmeotcp_queue + * @sk: The socket used by the NVMe-TCP queue + * @zerocopy: if this queue is used for zerocopy offload. + * @crc_rx: if this queue is used for CRC Rx offload. + * @ccid: ID of the current CC + * @ccsglidx: Index within the scatter-gather list (SGL) of the current CC + * @ccoff_inner: Current offset within the @ccsglidx element + * @priv: mlx5e netdev priv + * @inv_done: invalidate callback of the nvme tcp driver + * @after_resync_cqe: indicate if resync occurred + */ +struct mlx5e_nvmeotcp_queue { + struct ulp_ddp_ctx ulp_ddp_ctx; + struct mlx5_flow_handle *fh; + int tirn; + int id; + u32 size; + struct mlx5e_nvmeotcp_sq *sq; + spinlock_t nvmeotcp_icosq_lock; + struct nvmeotcp_queue_entry *ccid_table; + u32 tag_buf_table_id; + struct rhash_head hash; + refcount_t ref_count; + bool dgst; + int pda; + u32 ccid_gen; + u32 max_klms_per_wqe; + u32 channel_ix; + struct sock *sk; + bool zerocopy; + bool crc_rx; + + /* current ccid fields */ + off_t ccoff; + int ccid; + int ccsglidx; + int ccoff_inner; + + /* for ddp invalidate flow */ + struct mlx5e_priv *priv; + + /* for flow_steering flow */ + struct completion done; + /* for MASK HW resync cqe */ + bool after_resync_cqe; +}; + +struct mlx5e_nvmeotcp { + struct ida queue_ids; + struct rhashtable queue_hash; + bool enable; + bool crc_rx_enable; +}; + +void mlx5e_nvmeotcp_build_netdev(struct mlx5e_priv *priv); +int mlx5e_nvmeotcp_init(struct mlx5e_priv *priv); +int set_feature_nvme_tcp(struct net_device *netdev, bool enable); +int set_feature_nvme_tcp_crc(struct net_device *netdev, bool enable); +void mlx5e_nvmeotcp_cleanup(struct mlx5e_priv *priv); +int mlx5e_nvmeotcp_init_rx(struct mlx5e_priv *priv); +void mlx5e_nvmeotcp_cleanup_rx(struct mlx5e_priv *priv); +#else + +static inline void mlx5e_nvmeotcp_build_netdev(struct mlx5e_priv *priv) { } +static inline int mlx5e_nvmeotcp_init(struct mlx5e_priv *priv) { return 0; } +static inline void mlx5e_nvmeotcp_cleanup(struct mlx5e_priv *priv) { } +static inline int set_feature_nvme_tcp(struct net_device *netdev, bool enable) { return 0; } +static inline int set_feature_nvme_tcp_crc(struct net_device *netdev, bool enable) { return 0; } +static inline int mlx5e_nvmeotcp_init_rx(struct mlx5e_priv *priv) { return 0; } +static inline void mlx5e_nvmeotcp_cleanup_rx(struct mlx5e_priv *priv) { } +#endif +#endif /* __MLX5E_NVMEOTCP_H__ */ diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c index 414a73d16619..ecb12c7fdb7d 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c @@ -47,6 +47,7 @@ #include "en_accel/ipsec.h" #include "en_accel/en_accel.h" #include "en_accel/tls.h" +#include "en_accel/nvmeotcp.h" #include "accel/ipsec.h" #include "accel/tls.h" #include "lib/vxlan.h" @@ -2007,6 +2008,10 @@ static int mlx5e_open_channel(struct mlx5e_priv *priv, int ix, c->aff_mask = irq_get_effective_affinity_mask(irq); c->lag_port = mlx5e_enumerate_lag_port(priv->mdev, ix); +#ifdef CONFIG_MLX5_EN_NVMEOTCP + INIT_LIST_HEAD(&c->list_nvmeotcpsq); + spin_lock_init(&c->nvmeotcp_icosq_lock); +#endif netif_napi_add(netdev, &c->napi, mlx5e_napi_poll, 64); err = mlx5e_open_queues(c, params, cparam); @@ -3820,6 +3825,9 @@ int mlx5e_set_features(struct net_device *netdev, netdev_features_t features) err |= MLX5E_HANDLE_FEATURE(NETIF_F_NTUPLE, set_feature_arfs); #endif err |= MLX5E_HANDLE_FEATURE(NETIF_F_HW_TLS_RX, mlx5e_ktls_set_feature_rx); +#ifdef CONFIG_MLX5_EN_NVMEOTCP + err |= MLX5E_HANDLE_FEATURE(NETIF_F_HW_ULP_DDP, set_feature_nvme_tcp); +#endif if (err) { netdev->features = oper_features; @@ -3858,6 +3866,17 @@ static netdev_features_t mlx5e_fix_features(struct net_device *netdev, features &= ~NETIF_F_RXHASH; if (netdev->features & NETIF_F_RXHASH) netdev_warn(netdev, "Disabling rxhash, not supported when CQE compress is active\n"); + + features &= ~NETIF_F_HW_ULP_DDP; + if (netdev->features & NETIF_F_HW_ULP_DDP) + netdev_warn(netdev, "Disabling tcp-ddp offload, not supported when CQE compress is active\n"); + + } + + if (netdev->features & NETIF_F_LRO) { + features &= ~NETIF_F_HW_ULP_DDP; + if (netdev->features & NETIF_F_HW_ULP_DDP) + netdev_warn(netdev, "Disabling tcp-ddp offload, not supported when LRO is active\n"); } if (mlx5e_is_uplink_rep(priv)) { @@ -4890,6 +4909,7 @@ static void mlx5e_build_nic_netdev(struct net_device *netdev) mlx5e_set_netdev_dev_addr(netdev); mlx5e_ipsec_build_netdev(priv); mlx5e_tls_build_netdev(priv); + mlx5e_nvmeotcp_build_netdev(priv); } void mlx5e_create_q_counters(struct mlx5e_priv *priv) @@ -4950,6 +4970,10 @@ static int mlx5e_nic_init(struct mlx5_core_dev *mdev, if (err) mlx5_core_err(mdev, "TLS initialization failed, %d\n", err); + err = mlx5e_nvmeotcp_init(priv); + if (err) + mlx5_core_err(mdev, "NVMEoTCP initialization failed, %d\n", err); + dl_port = mlx5e_devlink_get_dl_port(priv); if (dl_port->registered) mlx5e_health_create_reporters(priv); @@ -4963,6 +4987,8 @@ static void mlx5e_nic_cleanup(struct mlx5e_priv *priv) if (dl_port->registered) mlx5e_health_destroy_reporters(priv); + + mlx5e_nvmeotcp_cleanup(priv); mlx5e_tls_cleanup(priv); mlx5e_ipsec_cleanup(priv); } diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c index 833be29170a1..3fc11b71de67 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c @@ -37,6 +37,7 @@ #include "en/xsk/rx.h" #include "en/xsk/tx.h" #include "en_accel/ktls_txrx.h" +#include "en_accel/nvmeotcp.h" static inline bool mlx5e_channel_no_affinity_change(struct mlx5e_channel *c) { @@ -119,6 +120,10 @@ int mlx5e_napi_poll(struct napi_struct *napi, int budget) struct mlx5e_txqsq __rcu **qos_sqs; struct mlx5e_rq *xskrq = &c->xskrq; struct mlx5e_rq *rq = &c->rq; +#ifdef CONFIG_MLX5_EN_NVMEOTCP + struct mlx5e_nvmeotcp_sq *nvmeotcp_sq; + struct list_head *cur; +#endif bool aff_change = false; bool busy_xsk = false; bool busy = false; @@ -171,6 +176,12 @@ int mlx5e_napi_poll(struct napi_struct *napi, int budget) * queueing more WQEs and overflowing the async ICOSQ. */ clear_bit(MLX5E_SQ_STATE_PENDING_XSK_TX, &c->async_icosq.state); +#ifdef CONFIG_MLX5_EN_NVMEOTCP + list_for_each(cur, &c->list_nvmeotcpsq) { + nvmeotcp_sq = list_entry(cur, struct mlx5e_nvmeotcp_sq, list); + mlx5e_poll_ico_cq(&nvmeotcp_sq->icosq.cq); + } +#endif /* Keep after async ICOSQ CQ poll */ if (unlikely(mlx5e_ktls_rx_pending_resync_list(c, budget))) @@ -223,6 +234,12 @@ int mlx5e_napi_poll(struct napi_struct *napi, int budget) mlx5e_cq_arm(&rq->cq); mlx5e_cq_arm(&c->icosq.cq); mlx5e_cq_arm(&c->async_icosq.cq); +#ifdef CONFIG_MLX5_EN_NVMEOTCP + list_for_each(cur, &c->list_nvmeotcpsq) { + nvmeotcp_sq = list_entry(cur, struct mlx5e_nvmeotcp_sq, list); + mlx5e_cq_arm(&nvmeotcp_sq->icosq.cq); + } +#endif mlx5e_cq_arm(&c->xdpsq.cq); if (xsk_open) { diff --git a/drivers/net/ethernet/mellanox/mlx5/core/fw.c b/drivers/net/ethernet/mellanox/mlx5/core/fw.c index 016d26f809a5..a8a14c15a61f 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/fw.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/fw.c @@ -262,6 +262,12 @@ int mlx5_query_hca_caps(struct mlx5_core_dev *dev) return err; } + if (MLX5_CAP_GEN(dev, nvmeotcp)) { + err = mlx5_core_get_caps(dev, MLX5_CAP_DEV_NVMEOTCP); + if (err) + return err; + } + return 0; } From patchwork Thu Jul 22 11:03:02 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Boris Pismenny X-Patchwork-Id: 485106 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-20.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER, INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5CB86C6377D for ; Thu, 22 Jul 2021 11:05:40 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 3B4456121E for ; Thu, 22 Jul 2021 11:05:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231691AbhGVKZD (ORCPT ); Thu, 22 Jul 2021 06:25:03 -0400 Received: from mail-bn8nam08on2053.outbound.protection.outlook.com ([40.107.100.53]:43297 "EHLO NAM04-BN8-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S231747AbhGVKYo (ORCPT ); Thu, 22 Jul 2021 06:24:44 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=dcB0XpOQlcOyXnE8ycyuirGwQX3+Z6qochVfQGgkHK6Wtu9f2+W8sh7iW91QVYf2AKmfOkvDccWhiKfJSY6sHkwmz1Wn0qRr9I9DQQLWwTweukkPg6rlep6LlmcT4ZKjxm5l5MJmNyVIWMKOe/bnfMWIoraKP65kIYg6e+FoROAF5iJz4zw6wAQMt/RoR8l+Z3h57RP0W/NyHyMlR/43o5TO0NA4u24zOdfSlPzOT7uFvs1QTGsiT22epr+R2d5CYePkORIMkGmnbFXDCwE4mX1JExwcJBjOAgB0i6jJKiO9P5l/U0w1hvieGl3U5XgBUG896uIk7p7s30ieKCBA9w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=+ShtaS6eo8A2UqvC6wCE654NQk0YfniZ9oq5OWx6/J8=; b=DDhCnWVmCah1wi3ik6MVwugNfpboQxe4vt0sS+m5aJtG/m0GDHqrxEWOcYbB/jux10nWpLl/ZDMTTzVIaYBPd5OQ99RGjpLkWACSE7vN9xlzZHLxtTfmk9h5YIyd/tlINvhcNLbzE9x3EaQuWg46vFbAWHc3q413j1B9NaQqC+MxeGnBB51LRwWxuWOgId76RppoRsYuvYnnvZxJYtSf0YLG3nKZjjKo58Akf5NTTUcrmdaxz4kMG39m5jqOcp9aPji748DsxqKSxuC2kLUfbbvfInsdLPPPkPCuCL7Bm8JM8Ul2aNW6Qlk747+qhGLdpNeNCmlg8os45wYwXsdcFA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.36) smtp.rcpttodomain=marvell.com smtp.mailfrom=nvidia.com; dmarc=pass (p=quarantine sp=none pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=+ShtaS6eo8A2UqvC6wCE654NQk0YfniZ9oq5OWx6/J8=; b=tqYi3nuvuqXEeBlvbTgt7hD/OwbXpU9RE3SfXBfoTH52TRtW8Bb/NLhEyezvmWnXO3df5AqF9hJTnp+aKgEcAG/VVJENaRT21q9g989V+lD5fpc5yS/LtWU3ZGU044+3T0nyXViIXp7Pb3xN0EQFV2wyIoWZ5xduzeZNHRYlScq+UQu8wxbE/fw377qra5+IV9waHvkaKQ0ne9CzGKMEyXlwt6TOWju/bzPIyG1O2SnBlOUVi4zGvyYzKaRGoh0WQRLz57u2dz87WBqUbrPM2QCaq2lydxjRwS2gAeD9D4duOPqRz/j4HPLpWO2AAFY73cCSQ3913yUB5tOacw9qGQ== Received: from BN6PR12CA0033.namprd12.prod.outlook.com (2603:10b6:405:70::19) by CH0PR12MB5043.namprd12.prod.outlook.com (2603:10b6:610:e2::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4331.21; Thu, 22 Jul 2021 11:05:17 +0000 Received: from BN8NAM11FT026.eop-nam11.prod.protection.outlook.com (2603:10b6:405:70:cafe::f2) by BN6PR12CA0033.outlook.office365.com (2603:10b6:405:70::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4352.26 via Frontend Transport; Thu, 22 Jul 2021 11:05:17 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.36) smtp.mailfrom=nvidia.com; marvell.com; dkim=none (message not signed) header.d=none; marvell.com; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.36 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.36; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.36) by BN8NAM11FT026.mail.protection.outlook.com (10.13.177.51) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4352.24 via Frontend Transport; Thu, 22 Jul 2021 11:05:17 +0000 Received: from HQMAIL107.nvidia.com (172.20.187.13) by HQMAIL101.nvidia.com (172.20.187.10) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Thu, 22 Jul 2021 11:05:16 +0000 Received: from vdi.nvidia.com (172.20.187.5) by mail.nvidia.com (172.20.187.13) with Microsoft SMTP Server id 15.0.1497.2 via Frontend Transport; Thu, 22 Jul 2021 11:05:12 +0000 From: Boris Pismenny To: , , , , , , , , , , CC: , , , , , , Boris Pismenny , Or Gerlitz , Yoray Zack Subject: [PATCH v5 net-next 13/36] net/mlx5e: KLM UMR helper macros Date: Thu, 22 Jul 2021 14:03:02 +0300 Message-ID: <20210722110325.371-14-borisp@nvidia.com> X-Mailer: git-send-email 2.24.1 In-Reply-To: <20210722110325.371-1-borisp@nvidia.com> References: <20210722110325.371-1-borisp@nvidia.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: fcfeca52-a895-4e25-6fdd-08d94d0096e2 X-MS-TrafficTypeDiagnostic: CH0PR12MB5043: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:439; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: +8ixLzW051jwRcdhKIbeEwdCNE4dn3bnMEqFADKqI8dQk9N0HKhxFzOoS+3uv2TBNv91GG89AThvbBQHtMWBr/T2qpphhyZTDtoL+/WPIuTq1Ou+EF0ti871LYXpsku7kPtNWvTJoE+j/q+GQ1h7nrulKyt4cT2d6rwHhOhkXLTm/XBj/XxwHBrz6a94fcM6iM+MzNmpF+fLACvlc44pXd+HFi00qFZ7VPxOtRzdjrFxcaZOIhAvSepZOu//E2prVu34dEu1N4wR6+y3OyquYLZDNUTBw3mWyYobBc3IBXqgV6ePhfnR4fh1WloF7Wf7QL2wA3AxYAFovAUvuk0FMgzf7AHRZvC8GbjNJoCdZB/0oGsX94ec62iw7rHPNKDLFAYAG0UW4ed0bBuTyXlTVynOBgagKjMyPiur/x5bt3dJb8go00D5Mcrs2x68xurzSWNH1fyW6uonrqIj/WUf2GD8B984+khXPcKY68lN7kWh68AoDyIEArLW2ymyveQl/qe5rc+L0rlFQtFt+9ChXwFYzDT5tMveILByWs4/TlHNA2NDUf8eD+jQwlY6eMTJFfswolb6rj5ESTTICNF3QfiH/8EnYgnrszDhJMAYgZuvpTRHUn10w9VKOjQWYDdsotTrr3ZepJwm1MvXmo3AXsysy/A1PvG8zhwXILJ/eMx/y5KOTh/tIwfklmGypO0lhmkMkck7BQqo37hs1yX7gn6avr5sbnzVsxT49qAAtmA= X-Forefront-Antispam-Report: CIP:216.228.112.36; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:schybrid05.nvidia.com; CAT:NONE; SFS:(4636009)(46966006)(36840700001)(107886003)(186003)(8676002)(82310400003)(86362001)(36906005)(47076005)(1076003)(7636003)(426003)(2616005)(356005)(54906003)(70206006)(70586007)(316002)(508600001)(336012)(7696005)(36860700001)(7416002)(36756003)(6666004)(5660300002)(110136005)(26005)(4326008)(8936002)(921005)(2906002); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Jul 2021 11:05:17.4817 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: fcfeca52-a895-4e25-6fdd-08d94d0096e2 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.112.36]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BN8NAM11FT026.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH0PR12MB5043 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Ben Ben-Ishay Add helper macros for posting KLM UMR WQE. Signed-off-by: Boris Pismenny Signed-off-by: Ben Ben-Ishay Signed-off-by: Or Gerlitz Signed-off-by: Yoray Zack --- drivers/net/ethernet/mellanox/mlx5/core/en.h | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en.h b/drivers/net/ethernet/mellanox/mlx5/core/en.h index 1233ebcf311b..5bc38002d136 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en.h +++ b/drivers/net/ethernet/mellanox/mlx5/core/en.h @@ -155,6 +155,24 @@ struct page_pool; #define MLX5E_UMR_WQEBBS \ (DIV_ROUND_UP(MLX5E_UMR_WQE_INLINE_SZ, MLX5_SEND_WQE_BB)) +#define KLM_ALIGNMENT 4 +#define MLX5E_KLM_UMR_WQE_SZ(sgl_len)\ + (sizeof(struct mlx5e_umr_wqe) +\ + (sizeof(struct mlx5_klm) * (sgl_len))) + +#define MLX5E_KLM_UMR_WQEBBS(sgl_len)\ + (DIV_ROUND_UP(MLX5E_KLM_UMR_WQE_SZ(sgl_len), MLX5_SEND_WQE_BB)) + +#define MLX5E_KLM_UMR_DS_CNT(sgl_len)\ + DIV_ROUND_UP(MLX5E_KLM_UMR_WQE_SZ(sgl_len), MLX5_SEND_WQE_DS) + +#define MLX5E_MAX_KLM_ENTRIES_PER_WQE(wqe_size)\ + (((wqe_size) - sizeof(struct mlx5e_umr_wqe)) / sizeof(struct mlx5_klm)) + +#define MLX5E_KLM_ENTRIES_PER_WQE(wqe_size)\ + (MLX5E_MAX_KLM_ENTRIES_PER_WQE(wqe_size) -\ + (MLX5E_MAX_KLM_ENTRIES_PER_WQE(wqe_size) % KLM_ALIGNMENT)) + #define MLX5E_MSG_LEVEL NETIF_MSG_LINK #define mlx5e_dbg(mlevel, priv, format, ...) \ From patchwork Thu Jul 22 11:03:03 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Boris Pismenny X-Patchwork-Id: 484319 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-20.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER, INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 48CD1C6377D for ; Thu, 22 Jul 2021 11:05:43 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 284386120D for ; Thu, 22 Jul 2021 11:05:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231675AbhGVKZG (ORCPT ); Thu, 22 Jul 2021 06:25:06 -0400 Received: from mail-dm6nam11on2051.outbound.protection.outlook.com ([40.107.223.51]:65409 "EHLO NAM11-DM6-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S231751AbhGVKYy (ORCPT ); Thu, 22 Jul 2021 06:24:54 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=GW7iU2lJakEqMd4qwY58nCV1fsdQULH9K8NkmnHDRE/UnIqbFhoAtyq4gbCv3OVSJIKTlfKUyCWxJ9P7Htsps7UcSvkYwttV3poY7oPgS5Fjj6//n3KEqnDKY6/jaRcVWLbnFrFfFm+y2q8uOiTfc+N1ICB9e7s8jvQkVyxr+QXt4VNEK7F7NL9Yx07iK139QR1/PdWTuUD9dMDuugzO3yS0nPdf7w/7cZ87eBN33xAHxhLjIEAwpX2pNbRyOej22NVA+G4DneRUYyr6lJBM6fPoai9glzHHWitQ3qW8LUQ2u1e17x4afueo8JCqkrq/TsFpqb6gnuazMStITNGqlw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=WL2JDoQ89OGPFn6abuFUKjgj2hi8qRpEuHy0HP4KAzw=; b=cz9TCUjOO82U4P9IZZ0pU2iVOw5AYIpGoDi0kN5BH0p3IStyaKbx/NO/1kgbGOoLkHvBB0lY8LI6TtWqntym6Xi9BVLQSMqFVsahz/h9ZFaIA0ESTjfhNggtkMvHTkbW44lOtsgiJVhwwzjMrtQVOsETwWTyBUl5b8MJQJMok3gbYfsj4Ebou9LmVY0fHmlHLqxPrCFs8AED5BdZGG57Axr2zBbDFa8Z1RGWp669ClxCeCHWMsbXBZe7IlsouGbXIrn1eXOtx0HzoUsFcr2tefgnqTIO/HD3fKs1MB8SqOvw4zsnmTyuSavSNTiWQfKnJAKBd3ZiTMAiiPrLbSfa2g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.32) smtp.rcpttodomain=marvell.com smtp.mailfrom=nvidia.com; dmarc=pass (p=quarantine sp=none pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=WL2JDoQ89OGPFn6abuFUKjgj2hi8qRpEuHy0HP4KAzw=; b=cajbvZ1MhXrwwx3Rn3IMFUTV8fPEUfbcmkvhReGmlRA1qi9O0W2TqjV+LdWoqEmHGBDQNkNutyGWvbaus+Twgr+G7jEk4mxgYyhWDShyj53ExaCNSIHY4WukAvPtqwcEsIqRsG/RT4ctQhb9UK6WIpDKh1b0vPjp/rxGjJhweRrjbh1bzzKCnjx+67BC5Ddgn2DaLw5VBLPWIbp4TpktMxtgmcIIGuoyygP9dcTrg4VD0mAX75cKqUyP1uwThgcJITFpFqbsfUnUWUtUNV0oK6VpygE4pYVWQNdRJrkOrXPzNwAzxs67+fV1zOrNYjwVuHkagJTK6c0ANFZBmhnIeQ== Received: from DM3PR14CA0144.namprd14.prod.outlook.com (2603:10b6:0:53::28) by SN6PR12MB2687.namprd12.prod.outlook.com (2603:10b6:805:73::26) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4331.26; Thu, 22 Jul 2021 11:05:22 +0000 Received: from DM6NAM11FT035.eop-nam11.prod.protection.outlook.com (2603:10b6:0:53:cafe::c7) by DM3PR14CA0144.outlook.office365.com (2603:10b6:0:53::28) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4352.25 via Frontend Transport; Thu, 22 Jul 2021 11:05:22 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.32) smtp.mailfrom=nvidia.com; marvell.com; dkim=none (message not signed) header.d=none; marvell.com; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.32 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.32; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.32) by DM6NAM11FT035.mail.protection.outlook.com (10.13.172.100) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4352.24 via Frontend Transport; Thu, 22 Jul 2021 11:05:22 +0000 Received: from HQMAIL107.nvidia.com (172.20.187.13) by HQMAIL109.nvidia.com (172.20.187.15) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Thu, 22 Jul 2021 04:05:21 -0700 Received: from vdi.nvidia.com (172.20.187.5) by mail.nvidia.com (172.20.187.13) with Microsoft SMTP Server id 15.0.1497.2 via Frontend Transport; Thu, 22 Jul 2021 11:05:17 +0000 From: Boris Pismenny To: , , , , , , , , , , CC: , , , , , , Boris Pismenny , Or Gerlitz , Yoray Zack Subject: [PATCH v5 net-next 14/36] net/mlx5e: NVMEoTCP use KLM UMRs Date: Thu, 22 Jul 2021 14:03:03 +0300 Message-ID: <20210722110325.371-15-borisp@nvidia.com> X-Mailer: git-send-email 2.24.1 In-Reply-To: <20210722110325.371-1-borisp@nvidia.com> References: <20210722110325.371-1-borisp@nvidia.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: feb18609-960d-4230-8464-08d94d0099c3 X-MS-TrafficTypeDiagnostic: SN6PR12MB2687: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:2887; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: zGDFobRx6VM8KD+iBQfbGcPgmpukjRk9tOoX2ISxJLDwmE8zEvyWDQ93flsT+3mmcnS7bCV3yVje4Ei+aN48l9tmhnPjRi6CjobcH1VnP5qmT2KCdP2UoTeEKi20qhS3gZFbA2qFt9QzGHIr867lFqvOlfgrxcG8tnW0ho6QuI8d1064mkrPq1SaFsc9CJnmNojKVsh8y/EibDSpfAYDudS1tbQvAjKaB1GpmXM9Uy1vE7ReWbXMceAQu0fKRLLH7noEbkVAUZQ/6OfQcO5tFoXzpYLA3AtURUdOp5qAXmBHlAXej9TrSTFWAenG8XsPiVuVT8J96B3LVXkKw4CHUjuXdm+oF/oBF2CHatojoangM2EJ4/zOYAqhjCLJ3Fle1AHjEniR3wjBYBQjanXNOn9/490usbCA4IO13eNTbD69DXVtw7UJKZ6d3XfiG09PwlZtlTRxiwoueyiZUzMDKQ6dOl6hynJ4T6iNv0M6fkmZZWGTViZvBZn5a+moYVJlpg+JJtavJ58evDGNbvD77rif7mmHQuJ5DdvNB8k9WkQt+iqF+8Mv0jEMuFM/d/o5t+sQ4iPFxBsjvSNZR32vK5Dmly0iSpZ0hp28KPGXXE6uuRHXBcDUGstdR9hHEoLmBZICU1ahlmxl6+imgrBts+crIGa3SvCPIVmkmFolma+1NPkXOnUCIvJaQV3oHPX/CqXwabEsqk/iaaY5B3SY5CvPywoGG1+6hCie9rKuneY= X-Forefront-Antispam-Report: CIP:216.228.112.32; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:schybrid01.nvidia.com; CAT:NONE; SFS:(4636009)(346002)(376002)(39860400002)(396003)(136003)(46966006)(36840700001)(6666004)(426003)(86362001)(4326008)(70586007)(36860700001)(26005)(478600001)(5660300002)(921005)(2906002)(47076005)(82310400003)(70206006)(107886003)(1076003)(356005)(7636003)(7416002)(186003)(83380400001)(8676002)(36756003)(7696005)(82740400003)(316002)(8936002)(336012)(54906003)(110136005)(2616005); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Jul 2021 11:05:22.3592 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: feb18609-960d-4230-8464-08d94d0099c3 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.112.32]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DM6NAM11FT035.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN6PR12MB2687 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Ben Ben-Ishay NVMEoTCP offload uses buffer registration for ddp operation, every request comprises from SG list that might have elements with size > 4K, thus the appropriate way to perform buffer registration is with KLM UMRs. Signed-off-by: Boris Pismenny Signed-off-by: Ben Ben-Ishay Signed-off-by: Or Gerlitz Signed-off-by: Yoray Zack --- drivers/net/ethernet/mellanox/mlx5/core/en.h | 5 +- .../net/ethernet/mellanox/mlx5/core/en/txrx.h | 3 + .../mellanox/mlx5/core/en_accel/nvmeotcp.c | 116 ++++++++++++++++++ .../mlx5/core/en_accel/nvmeotcp_utils.h | 12 ++ .../net/ethernet/mellanox/mlx5/core/en_rx.c | 12 +- 5 files changed, 145 insertions(+), 3 deletions(-) create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp_utils.h diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en.h b/drivers/net/ethernet/mellanox/mlx5/core/en.h index 5bc38002d136..d88ecd04d5da 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en.h +++ b/drivers/net/ethernet/mellanox/mlx5/core/en.h @@ -238,7 +238,10 @@ struct mlx5e_umr_wqe { struct mlx5_wqe_ctrl_seg ctrl; struct mlx5_wqe_umr_ctrl_seg uctrl; struct mlx5_mkey_seg mkc; - struct mlx5_mtt inline_mtts[0]; + union { + struct mlx5_mtt inline_mtts[0]; + struct mlx5_klm inline_klms[0]; + }; }; extern const char mlx5e_self_tests[][ETH_GSTRING_LEN]; diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h b/drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h index 055c3bc23733..529745aac7e8 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h +++ b/drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h @@ -41,6 +41,9 @@ enum mlx5e_icosq_wqe_type { MLX5E_ICOSQ_WQE_SET_PSV_TLS, MLX5E_ICOSQ_WQE_GET_PSV_TLS, #endif +#ifdef CONFIG_MLX5_EN_NVMEOTCP + MLX5E_ICOSQ_WQE_UMR_NVME_TCP, +#endif }; /* General */ diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c index 04e88042b243..7fc3b13b1b35 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c @@ -4,6 +4,7 @@ #include #include #include "en_accel/nvmeotcp.h" +#include "en_accel/nvmeotcp_utils.h" #include "en_accel/fs_tcp.h" #include "en/txrx.h" @@ -19,6 +20,121 @@ static const struct rhashtable_params rhash_queues = { .max_size = MAX_NVMEOTCP_QUEUES, }; +static void +fill_nvmeotcp_klm_wqe(struct mlx5e_nvmeotcp_queue *queue, + struct mlx5e_umr_wqe *wqe, u16 ccid, u32 klm_entries, + u16 klm_offset) +{ + struct scatterlist *sgl_mkey; + u32 lkey, i; + + lkey = queue->priv->mdev->mlx5e_res.mkey.key; + for (i = 0; i < klm_entries; i++) { + sgl_mkey = &queue->ccid_table[ccid].sgl[i + klm_offset]; + wqe->inline_klms[i].bcount = cpu_to_be32(sgl_mkey->length); + wqe->inline_klms[i].key = cpu_to_be32(lkey); + wqe->inline_klms[i].va = cpu_to_be64(sgl_mkey->dma_address); + } + + for (; i < ALIGN(klm_entries, KLM_ALIGNMENT); i++) { + wqe->inline_klms[i].bcount = 0; + wqe->inline_klms[i].key = 0; + wqe->inline_klms[i].va = 0; + } +} + +static void +build_nvmeotcp_klm_umr(struct mlx5e_nvmeotcp_queue *queue, + struct mlx5e_umr_wqe *wqe, u16 ccid, int klm_entries, + u32 klm_offset, u32 len) +{ + u32 id = queue->ccid_table[ccid].klm_mkey.key; + struct mlx5_wqe_umr_ctrl_seg *ucseg = &wqe->uctrl; + struct mlx5_wqe_ctrl_seg *cseg = &wqe->ctrl; + struct mlx5_mkey_seg *mkc = &wqe->mkc; + + u32 sqn = queue->sq->icosq.sqn; + u16 pc = queue->sq->icosq.pc; + + cseg->opmod_idx_opcode = cpu_to_be32((pc << MLX5_WQE_CTRL_WQE_INDEX_SHIFT) | + MLX5_OPCODE_UMR); + cseg->qpn_ds = cpu_to_be32((sqn << MLX5_WQE_CTRL_QPN_SHIFT) | + MLX5E_KLM_UMR_DS_CNT(ALIGN(klm_entries, KLM_ALIGNMENT))); + cseg->general_id = cpu_to_be32(id); + + if (!klm_offset) { + ucseg->mkey_mask |= cpu_to_be64(MLX5_MKEY_MASK_XLT_OCT_SIZE | + MLX5_MKEY_MASK_LEN | MLX5_MKEY_MASK_FREE); + mkc->xlt_oct_size = cpu_to_be32(ALIGN(len, KLM_ALIGNMENT)); + mkc->len = cpu_to_be64(queue->ccid_table[ccid].size); + } + + ucseg->flags = MLX5_UMR_INLINE | MLX5_UMR_TRANSLATION_OFFSET_EN; + ucseg->xlt_octowords = cpu_to_be16(ALIGN(klm_entries, KLM_ALIGNMENT)); + ucseg->xlt_offset = cpu_to_be16(klm_offset); + fill_nvmeotcp_klm_wqe(queue, wqe, ccid, klm_entries, klm_offset); +} + +static void +mlx5e_nvmeotcp_fill_wi(struct mlx5e_nvmeotcp_queue *nvmeotcp_queue, + struct mlx5e_icosq *sq, u32 wqe_bbs, u16 pi) +{ + struct mlx5e_icosq_wqe_info *wi = &sq->db.wqe_info[pi]; + + wi->num_wqebbs = wqe_bbs; + wi->wqe_type = MLX5E_ICOSQ_WQE_UMR_NVME_TCP; +} + +static void +post_klm_wqe(struct mlx5e_nvmeotcp_queue *queue, + u16 ccid, + u32 klm_length, + u32 *klm_offset) +{ + struct mlx5e_icosq *sq = &queue->sq->icosq; + u32 wqe_bbs, cur_klm_entries; + struct mlx5e_umr_wqe *wqe; + u16 pi, wqe_sz; + + cur_klm_entries = min_t(int, queue->max_klms_per_wqe, + klm_length - *klm_offset); + wqe_sz = MLX5E_KLM_UMR_WQE_SZ(ALIGN(cur_klm_entries, KLM_ALIGNMENT)); + wqe_bbs = DIV_ROUND_UP(wqe_sz, MLX5_SEND_WQE_BB); + pi = mlx5e_icosq_get_next_pi(sq, wqe_bbs); + wqe = MLX5E_NVMEOTCP_FETCH_KLM_WQE(sq, pi); + mlx5e_nvmeotcp_fill_wi(queue, sq, wqe_bbs, pi); + build_nvmeotcp_klm_umr(queue, wqe, ccid, cur_klm_entries, *klm_offset, + klm_length); + *klm_offset += cur_klm_entries; + sq->pc += wqe_bbs; + sq->doorbell_cseg = &wqe->ctrl; +} + +static int +mlx5e_nvmeotcp_post_klm_wqe(struct mlx5e_nvmeotcp_queue *queue, + u16 ccid, + u32 klm_length) +{ + u32 klm_offset = 0, wqes, wqe_sz, max_wqe_bbs, i, room; + struct mlx5e_icosq *sq = &queue->sq->icosq; + + /* TODO: set stricter wqe_sz; using max for now */ + wqes = DIV_ROUND_UP(klm_length, queue->max_klms_per_wqe); + wqe_sz = MLX5E_KLM_UMR_WQE_SZ(queue->max_klms_per_wqe); + + max_wqe_bbs = DIV_ROUND_UP(wqe_sz, MLX5_SEND_WQE_BB); + + room = mlx5e_stop_room_for_wqe(max_wqe_bbs) * wqes; + if (unlikely(!mlx5e_wqc_has_room_for(&sq->wq, sq->cc, sq->pc, room))) + return -ENOSPC; + + for (i = 0; i < wqes; i++) + post_klm_wqe(queue, ccid, klm_length, &klm_offset); + + mlx5e_notify_hw(&sq->wq, sq->pc, sq->uar_map, sq->doorbell_cseg); + return 0; +} + static int mlx5e_nvmeotcp_offload_limits(struct net_device *netdev, struct ulp_ddp_limits *limits) diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp_utils.h b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp_utils.h new file mode 100644 index 000000000000..329e114d6571 --- /dev/null +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp_utils.h @@ -0,0 +1,12 @@ +/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */ +/* Copyright (c) 2021 Mellanox Technologies. */ +#ifndef __MLX5E_NVMEOTCP_UTILS_H__ +#define __MLX5E_NVMEOTCP_UTILS_H__ + +#include "en.h" + +#define MLX5E_NVMEOTCP_FETCH_KLM_WQE(sq, pi) \ + ((struct mlx5e_umr_wqe *)\ + mlx5e_fetch_wqe(&(sq)->wq, pi, sizeof(struct mlx5e_umr_wqe))) + +#endif /* __MLX5E_NVMEOTCP_UTILS_H__ */ diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c index 3c65fd0bcf31..9d821facbca4 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c @@ -615,16 +615,20 @@ void mlx5e_free_icosq_descs(struct mlx5e_icosq *sq) ci = mlx5_wq_cyc_ctr2ix(&sq->wq, sqcc); wi = &sq->db.wqe_info[ci]; sqcc += wi->num_wqebbs; -#ifdef CONFIG_MLX5_EN_TLS switch (wi->wqe_type) { +#ifdef CONFIG_MLX5_EN_TLS case MLX5E_ICOSQ_WQE_SET_PSV_TLS: mlx5e_ktls_handle_ctx_completion(wi); break; case MLX5E_ICOSQ_WQE_GET_PSV_TLS: mlx5e_ktls_handle_get_psv_completion(wi, sq); break; - } #endif +#ifdef CONFIG_MLX5_EN_NVMEOTCP + case MLX5E_ICOSQ_WQE_UMR_NVME_TCP: + break; +#endif + } } sq->cc = sqcc; } @@ -694,6 +698,10 @@ int mlx5e_poll_ico_cq(struct mlx5e_cq *cq) case MLX5E_ICOSQ_WQE_GET_PSV_TLS: mlx5e_ktls_handle_get_psv_completion(wi, sq); break; +#endif +#ifdef CONFIG_MLX5_EN_NVMEOTCP + case MLX5E_ICOSQ_WQE_UMR_NVME_TCP: + break; #endif default: netdev_WARN_ONCE(cq->netdev, From patchwork Thu Jul 22 11:03:04 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Boris Pismenny X-Patchwork-Id: 485105 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-20.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER, INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 949CEC63797 for ; Thu, 22 Jul 2021 11:05:45 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 770716120D for ; Thu, 22 Jul 2021 11:05:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231773AbhGVKZI (ORCPT ); Thu, 22 Jul 2021 06:25:08 -0400 Received: from mail-dm6nam12on2050.outbound.protection.outlook.com ([40.107.243.50]:25995 "EHLO NAM12-DM6-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S231778AbhGVKYy (ORCPT ); Thu, 22 Jul 2021 06:24:54 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=IQbCisyE8pRQCauVsXD8TCU0q/eml/mAwiaz5trWb134KA/qE+F1Bnyo9+1f3ZflNUEQn/AdOQbmU24wyd83pmyGQP2oJS2PN1f1i7UNdHoroQTCFY6kSnsArQCq89oV3Skwa/NtkAiUMQWlw61rmcn6dAm6E5W8Z62unCWFh5qEC0H/NgbMG1k7seX/ZUYMzIaioYP9qKQmA7LbZ3Gm7fzrlU4jhayvZ+P3zesiquaOFpLpdgwRP14oATXkB/WD+kZCjuYNvsmcsqBy25a16VL9tww5RWysT5yN3wAcFVh7H5YIFdtH7Arvdxu/vU61s/+8+0OWfzAd4WHf3UC8mg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=CdG1clGFZ9YUcts9jPUOunuPs6AFQzoOVhv7flVK/t4=; b=MaLhLenomx6kBvPmESpTPUOMWwqj5Cg+Fnk4V9JRVz1A4Uqut8JwZpgRjbE7wWbG7fIGl5o2UONKjMtwguAK+ea6t8HxkhZOsVqGK1hNeS4vKcsjAHTiQGLcjxtoyttOLZtiUM7bPHeDIiYC1PkyruGiuB/5RZeJYZB3QbsvN4lqTnGNgEETjQpOkPfjJ84Ny7uKSNINRnUJg3wimN65d6l5qTrfg62EVlH1heIkKWicqxil3DdpI/4fWNNoR482K3yN1Shua3oSoh7G/0syrdw/qi0oZ9mv0LrNymyKXAV+fFpnb0tictbNnTa9HnUyMCnLBNufPHdx1WXEFLIk4Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.35) smtp.rcpttodomain=marvell.com smtp.mailfrom=nvidia.com; dmarc=pass (p=quarantine sp=none pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=CdG1clGFZ9YUcts9jPUOunuPs6AFQzoOVhv7flVK/t4=; b=HNPi7pNlj18KbgYn+cBs2lhSjYfeE/Ag6xbDgCfdfct/mQN4XS9v352LIzcrP7/nQkDlXGfYHu/DRr2dq69xx9QXNYXxW+j7czzes6VN6eOKs+C/DhT2TkEtTGcZvW0Vn3+HOi7hUma1a3h0nyT68esDbQI6cJd9grkvglHOtlAeivu8RHsMuvqGCO25sCXqVQJC0T6Ys1+r3Z196pkERlxAhfM+1zWCyjFGA2CF/NGmsaN3efC92YCyudHn3QPvtinvWHuIVho59bDoHfNIaHdBROBWNXpcOMCDKVqNH/HDOM8LooC7vIpZOrczgJ+I57CkUetf+hmkC8xLldBq4w== Received: from DM5PR07CA0167.namprd07.prod.outlook.com (2603:10b6:3:ee::33) by MN2PR12MB3087.namprd12.prod.outlook.com (2603:10b6:208:d2::32) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4331.31; Thu, 22 Jul 2021 11:05:27 +0000 Received: from DM6NAM11FT055.eop-nam11.prod.protection.outlook.com (2603:10b6:3:ee:cafe::8c) by DM5PR07CA0167.outlook.office365.com (2603:10b6:3:ee::33) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4352.26 via Frontend Transport; Thu, 22 Jul 2021 11:05:27 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.35) smtp.mailfrom=nvidia.com; marvell.com; dkim=none (message not signed) header.d=none; marvell.com; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.35 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.35; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.35) by DM6NAM11FT055.mail.protection.outlook.com (10.13.173.103) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4352.24 via Frontend Transport; Thu, 22 Jul 2021 11:05:27 +0000 Received: from HQMAIL107.nvidia.com (172.20.187.13) by HQMAIL111.nvidia.com (172.20.187.18) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Thu, 22 Jul 2021 11:05:26 +0000 Received: from vdi.nvidia.com (172.20.187.5) by mail.nvidia.com (172.20.187.13) with Microsoft SMTP Server id 15.0.1497.2 via Frontend Transport; Thu, 22 Jul 2021 11:05:21 +0000 From: Boris Pismenny To: , , , , , , , , , , CC: , , , , , , Boris Pismenny , Or Gerlitz , Yoray Zack Subject: [PATCH v5 net-next 15/36] net/mlx5e: NVMEoTCP queue init/teardown Date: Thu, 22 Jul 2021 14:03:04 +0300 Message-ID: <20210722110325.371-16-borisp@nvidia.com> X-Mailer: git-send-email 2.24.1 In-Reply-To: <20210722110325.371-1-borisp@nvidia.com> References: <20210722110325.371-1-borisp@nvidia.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 027119c6-c578-4996-1900-08d94d009c8a X-MS-TrafficTypeDiagnostic: MN2PR12MB3087: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:6790; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: sTo4M3TNWzsD6sc4FT6A7RPtWE0YR2gL69xqmWNK+ThGd34Td8v1zTxMKzRKbdSgIyIvGyrNZ+MUQccshDDFZv/t2NPCU6MNOXInuP+b87xibXkd0bq/COffnI7h3yQZwVdax8Op9Jf/6KJUpzGisnrsgfh1okCztjjibg57fL4UVHTwDF/Q1sydOymOTwLBad0OmhXRUD++BnEEr4k8MpHfcIiwiSB6PmfBwYKtyw8zDnBCEH+OWmKlRZ9Tfaho+imTBTd2YSmeVkFqpQSGG6cz9Ix4B9AXA+5ocfDmhbBmuFwqRJhFi3SNnhTxinbEn4HFtNkUWz/4kem0sBDXbsHHUk0V/ByT8LUlOQa4mneSbj6oETh3pj/k/gazY027EBstHwnA67YK0+U7kWMIy8zwuJ0UfMukTxHG7ZyRx4QkSOMqmStSXU4pWidf913YYBXJKGWzZfImqFx6SjIkRe8O6G0es8AjSTCYNZpM6wMxxiuOcMoQ99EGWOesa2o6XEFSq3jU8k20XcZ51eIE+mozrfkKbqCAt0SUbv7lYv1I0MktjTYKBKProH7d5eaS9fpR1/ooJdMyB0AGFe5GM368h31C0Hn6Ui4sOSxlz0/eRAV1E81sLS7tUDpF9lSfY1jufhviGLhANBrSq97ThOxi2zmli0ZatQXCDmLRvMNOj/on4gtNMfjokt9iHrZviAFC/iDybbjZ6gSRgcshE6OgDNbroS7g7zaJ4ABqRtY= X-Forefront-Antispam-Report: CIP:216.228.112.35; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:schybrid04.nvidia.com; CAT:NONE; SFS:(4636009)(396003)(346002)(136003)(39860400002)(376002)(46966006)(36840700001)(336012)(54906003)(1076003)(26005)(36860700001)(7416002)(921005)(82310400003)(478600001)(70586007)(36756003)(107886003)(83380400001)(47076005)(70206006)(2616005)(2906002)(4326008)(7696005)(316002)(7636003)(82740400003)(356005)(86362001)(8676002)(8936002)(110136005)(5660300002)(36906005)(186003)(30864003)(426003)(6666004); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Jul 2021 11:05:27.0081 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 027119c6-c578-4996-1900-08d94d009c8a X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.112.35]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DM6NAM11FT055.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR12MB3087 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Ben Ben-Ishay When nvme-tcp establishes new connections, we allocate a hardware context to offload operations for this queue: - Use a separate TIR to identify the queue and maintain the HW context - Use a separate ICOSQ for maintain the HW context - Use a separate tag buffer for buffer registration - Maintain static and progress HW contexts by posting the proper WQEs at creation time, or upon resync Queue teardown will free the corresponding contexts. Signed-off-by: Boris Pismenny Signed-off-by: Ben Ben-Ishay Signed-off-by: Or Gerlitz Signed-off-by: Yoray Zack --- .../net/ethernet/mellanox/mlx5/core/en/txrx.h | 6 + .../mellanox/mlx5/core/en_accel/nvmeotcp.c | 663 +++++++++++++++++- .../mellanox/mlx5/core/en_accel/nvmeotcp.h | 4 + .../mlx5/core/en_accel/nvmeotcp_utils.h | 68 ++ .../net/ethernet/mellanox/mlx5/core/en_rx.c | 7 + 5 files changed, 723 insertions(+), 25 deletions(-) diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h b/drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h index 529745aac7e8..497c49f28d8a 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h +++ b/drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h @@ -43,6 +43,7 @@ enum mlx5e_icosq_wqe_type { #endif #ifdef CONFIG_MLX5_EN_NVMEOTCP MLX5E_ICOSQ_WQE_UMR_NVME_TCP, + MLX5E_ICOSQ_WQE_SET_PSV_NVME_TCP, #endif }; @@ -185,6 +186,11 @@ struct mlx5e_icosq_wqe_info { struct { struct mlx5e_ktls_rx_resync_buf *buf; } tls_get_params; +#endif +#ifdef CONFIG_MLX5_EN_NVMEOTCP + struct { + struct mlx5e_nvmeotcp_queue *queue; + } nvmeotcp_q; #endif }; }; diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c index 7fc3b13b1b35..2283b2a799f8 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c @@ -3,6 +3,7 @@ #include #include +#include #include "en_accel/nvmeotcp.h" #include "en_accel/nvmeotcp_utils.h" #include "en_accel/fs_tcp.h" @@ -20,35 +21,180 @@ static const struct rhashtable_params rhash_queues = { .max_size = MAX_NVMEOTCP_QUEUES, }; +#define MLX5_NVME_TCP_MAX_SEGMENTS 128 + +static u32 mlx5e_get_max_sgl(struct mlx5_core_dev *mdev) +{ + return min_t(u32, + MLX5_NVME_TCP_MAX_SEGMENTS, + 1 << MLX5_CAP_GEN(mdev, log_max_klm_list_size)); +} + +static void mlx5e_nvmeotcp_destroy_tir(struct mlx5e_priv *priv, int tirn) +{ + mlx5_core_destroy_tir(priv->mdev, tirn); +} + +static inline u32 +mlx5e_get_channel_ix_from_io_cpu(struct mlx5e_priv *priv, u32 io_cpu) +{ + int num_channels = priv->channels.params.num_channels; + u32 channel_ix = io_cpu; + + if (channel_ix >= num_channels) + channel_ix = channel_ix % num_channels; + + return channel_ix; +} + +static int mlx5e_nvmeotcp_create_tir(struct mlx5e_priv *priv, + struct sock *sk, + struct nvme_tcp_ddp_config *config, + struct mlx5e_nvmeotcp_queue *queue, + bool zerocopy, bool crc_rx) +{ + u32 rqtn = priv->direct_tir[queue->channel_ix].rqt.rqtn; + int err, inlen; + void *tirc; + u32 tirn; + u32 *in; + + inlen = MLX5_ST_SZ_BYTES(create_tir_in); + in = kvzalloc(inlen, GFP_KERNEL); + if (!in) + return -ENOMEM; + tirc = MLX5_ADDR_OF(create_tir_in, in, ctx); + MLX5_SET(tirc, tirc, disp_type, MLX5_TIRC_DISP_TYPE_INDIRECT); + MLX5_SET(tirc, tirc, rx_hash_fn, MLX5_RX_HASH_FN_INVERTED_XOR8); + MLX5_SET(tirc, tirc, indirect_table, rqtn); + MLX5_SET(tirc, tirc, transport_domain, priv->mdev->mlx5e_res.hw_objs.td.tdn); + if (zerocopy) { + MLX5_SET(tirc, tirc, nvmeotcp_zero_copy_en, 1); + MLX5_SET(tirc, tirc, nvmeotcp_tag_buffer_table_id, + queue->tag_buf_table_id); + } + + if (crc_rx) + MLX5_SET(tirc, tirc, nvmeotcp_crc_en, 1); + + MLX5_SET(tirc, tirc, self_lb_block, + MLX5_TIRC_SELF_LB_BLOCK_BLOCK_UNICAST | + MLX5_TIRC_SELF_LB_BLOCK_BLOCK_MULTICAST); + err = mlx5_core_create_tir(priv->mdev, in, &tirn); + + if (!err) + queue->tirn = tirn; + + kvfree(in); + return err; +} + +static +int mlx5e_create_nvmeotcp_tag_buf_table(struct mlx5_core_dev *mdev, + struct mlx5e_nvmeotcp_queue *queue, + u8 log_table_size) +{ + u32 in[MLX5_ST_SZ_DW(create_nvmeotcp_tag_buf_table_in)] = {}; + u32 out[MLX5_ST_SZ_DW(general_obj_out_cmd_hdr)]; + u64 general_obj_types; + void *obj; + int err; + + obj = MLX5_ADDR_OF(create_nvmeotcp_tag_buf_table_in, in, + nvmeotcp_tag_buf_table_obj); + + general_obj_types = MLX5_CAP_GEN_64(mdev, general_obj_types); + if (!(general_obj_types & + MLX5_HCA_CAP_GENERAL_OBJECT_TYPES_NVMEOTCP_TAG_BUFFER_TABLE)) + return -EINVAL; + + MLX5_SET(general_obj_in_cmd_hdr, in, opcode, + MLX5_CMD_OP_CREATE_GENERAL_OBJECT); + MLX5_SET(general_obj_in_cmd_hdr, in, obj_type, + MLX5_GENERAL_OBJECT_TYPES_NVMEOTCP_TAG_BUFFER_TABLE); + MLX5_SET(nvmeotcp_tag_buf_table_obj, obj, + log_tag_buffer_table_size, log_table_size); + + err = mlx5_cmd_exec(mdev, in, sizeof(in), out, sizeof(out)); + if (!err) + queue->tag_buf_table_id = MLX5_GET(general_obj_out_cmd_hdr, + out, obj_id); + return err; +} + +static +void mlx5_destroy_nvmeotcp_tag_buf_table(struct mlx5_core_dev *mdev, u32 uid) +{ + u32 in[MLX5_ST_SZ_DW(general_obj_in_cmd_hdr)] = {}; + u32 out[MLX5_ST_SZ_DW(general_obj_out_cmd_hdr)]; + + MLX5_SET(general_obj_in_cmd_hdr, in, opcode, + MLX5_CMD_OP_DESTROY_GENERAL_OBJECT); + MLX5_SET(general_obj_in_cmd_hdr, in, obj_type, + MLX5_GENERAL_OBJECT_TYPES_NVMEOTCP_TAG_BUFFER_TABLE); + MLX5_SET(general_obj_in_cmd_hdr, in, obj_id, uid); + + mlx5_cmd_exec(mdev, in, sizeof(in), out, sizeof(out)); +} + +#define MLX5_CTRL_SEGMENT_OPC_MOD_UMR_TIR_PARAMS 0x2 +#define MLX5_CTRL_SEGMENT_OPC_MOD_UMR_NVMEOTCP_TIR_STATIC_PARAMS 0x2 +#define MLX5_CTRL_SEGMENT_OPC_MOD_UMR_UMR 0x0 + +#define STATIC_PARAMS_DS_CNT \ + DIV_ROUND_UP(MLX5E_NVMEOTCP_STATIC_PARAMS_WQE_SZ, MLX5_SEND_WQE_DS) + +#define PROGRESS_PARAMS_DS_CNT \ + DIV_ROUND_UP(MLX5E_NVMEOTCP_PROGRESS_PARAMS_WQE_SZ, MLX5_SEND_WQE_DS) + +enum wqe_type { + KLM_UMR = 0, + BSF_KLM_UMR = 1, + SET_PSV_UMR = 2, + BSF_UMR = 3, +}; + static void fill_nvmeotcp_klm_wqe(struct mlx5e_nvmeotcp_queue *queue, struct mlx5e_umr_wqe *wqe, u16 ccid, u32 klm_entries, - u16 klm_offset) + u16 klm_offset, enum wqe_type klm_type) { struct scatterlist *sgl_mkey; u32 lkey, i; - lkey = queue->priv->mdev->mlx5e_res.mkey.key; - for (i = 0; i < klm_entries; i++) { - sgl_mkey = &queue->ccid_table[ccid].sgl[i + klm_offset]; - wqe->inline_klms[i].bcount = cpu_to_be32(sgl_mkey->length); - wqe->inline_klms[i].key = cpu_to_be32(lkey); - wqe->inline_klms[i].va = cpu_to_be64(sgl_mkey->dma_address); - } - - for (; i < ALIGN(klm_entries, KLM_ALIGNMENT); i++) { - wqe->inline_klms[i].bcount = 0; - wqe->inline_klms[i].key = 0; - wqe->inline_klms[i].va = 0; + if (klm_type == BSF_KLM_UMR) { + for (i = 0; i < klm_entries; i++) { + lkey = queue->ccid_table[i + klm_offset].klm_mkey.key; + wqe->inline_klms[i].bcount = cpu_to_be32(U32_MAX); + wqe->inline_klms[i].key = cpu_to_be32(lkey); + wqe->inline_klms[i].va = 0; + } + } else { + lkey = queue->priv->mdev->mlx5e_res.hw_objs.mkey.key; + for (i = 0; i < klm_entries; i++) { + sgl_mkey = &queue->ccid_table[ccid].sgl[i + klm_offset]; + wqe->inline_klms[i].bcount = cpu_to_be32(sgl_mkey->length); + wqe->inline_klms[i].key = cpu_to_be32(lkey); + wqe->inline_klms[i].va = cpu_to_be64(sgl_mkey->dma_address); + } + + for (; i < ALIGN(klm_entries, KLM_ALIGNMENT); i++) { + wqe->inline_klms[i].bcount = 0; + wqe->inline_klms[i].key = 0; + wqe->inline_klms[i].va = 0; + } } } static void build_nvmeotcp_klm_umr(struct mlx5e_nvmeotcp_queue *queue, struct mlx5e_umr_wqe *wqe, u16 ccid, int klm_entries, - u32 klm_offset, u32 len) + u32 klm_offset, u32 len, enum wqe_type klm_type) { - u32 id = queue->ccid_table[ccid].klm_mkey.key; + u32 id = (klm_type == KLM_UMR) ? queue->ccid_table[ccid].klm_mkey.key : + (queue->tirn << MLX5_WQE_CTRL_TIR_TIS_INDEX_SHIFT); + u8 opc_mod = (klm_type == KLM_UMR) ? MLX5_CTRL_SEGMENT_OPC_MOD_UMR_UMR : + MLX5_CTRL_SEGMENT_OPC_MOD_UMR_NVMEOTCP_TIR_STATIC_PARAMS; struct mlx5_wqe_umr_ctrl_seg *ucseg = &wqe->uctrl; struct mlx5_wqe_ctrl_seg *cseg = &wqe->ctrl; struct mlx5_mkey_seg *mkc = &wqe->mkc; @@ -57,36 +203,170 @@ build_nvmeotcp_klm_umr(struct mlx5e_nvmeotcp_queue *queue, u16 pc = queue->sq->icosq.pc; cseg->opmod_idx_opcode = cpu_to_be32((pc << MLX5_WQE_CTRL_WQE_INDEX_SHIFT) | - MLX5_OPCODE_UMR); + MLX5_OPCODE_UMR | (opc_mod) << 24); cseg->qpn_ds = cpu_to_be32((sqn << MLX5_WQE_CTRL_QPN_SHIFT) | MLX5E_KLM_UMR_DS_CNT(ALIGN(klm_entries, KLM_ALIGNMENT))); cseg->general_id = cpu_to_be32(id); - if (!klm_offset) { + if (klm_type == KLM_UMR && !klm_offset) { ucseg->mkey_mask |= cpu_to_be64(MLX5_MKEY_MASK_XLT_OCT_SIZE | MLX5_MKEY_MASK_LEN | MLX5_MKEY_MASK_FREE); mkc->xlt_oct_size = cpu_to_be32(ALIGN(len, KLM_ALIGNMENT)); mkc->len = cpu_to_be64(queue->ccid_table[ccid].size); } + ucseg->mkey_mask |= cpu_to_be64(MLX5_MKEY_MASK_FREE); ucseg->flags = MLX5_UMR_INLINE | MLX5_UMR_TRANSLATION_OFFSET_EN; ucseg->xlt_octowords = cpu_to_be16(ALIGN(klm_entries, KLM_ALIGNMENT)); ucseg->xlt_offset = cpu_to_be16(klm_offset); - fill_nvmeotcp_klm_wqe(queue, wqe, ccid, klm_entries, klm_offset); + fill_nvmeotcp_klm_wqe(queue, wqe, ccid, klm_entries, klm_offset, klm_type); +} + +static void +fill_nvmeotcp_progress_params(struct mlx5e_nvmeotcp_queue *queue, + struct mlx5_seg_nvmeotcp_progress_params *params, + u32 seq) +{ + void *ctx = params->ctx; + + params->tir_num = cpu_to_be32(queue->tirn); + + MLX5_SET(nvmeotcp_progress_params, ctx, + next_pdu_tcp_sn, seq); + MLX5_SET(nvmeotcp_progress_params, ctx, pdu_tracker_state, + MLX5E_NVMEOTCP_PROGRESS_PARAMS_PDU_TRACKER_STATE_START); +} + +void +build_nvmeotcp_progress_params(struct mlx5e_nvmeotcp_queue *queue, + struct mlx5e_set_nvmeotcp_progress_params_wqe *wqe, + u32 seq) +{ + struct mlx5_wqe_ctrl_seg *cseg = &wqe->ctrl; + u32 sqn = queue->sq->icosq.sqn; + u16 pc = queue->sq->icosq.pc; + u8 opc_mod; + + memset(wqe, 0, MLX5E_NVMEOTCP_PROGRESS_PARAMS_WQE_SZ); + opc_mod = MLX5_CTRL_SEGMENT_OPC_MOD_UMR_NVMEOTCP_TIR_PROGRESS_PARAMS; + cseg->opmod_idx_opcode = cpu_to_be32((pc << MLX5_WQE_CTRL_WQE_INDEX_SHIFT) | + MLX5_OPCODE_SET_PSV | (opc_mod << 24)); + cseg->qpn_ds = cpu_to_be32((sqn << MLX5_WQE_CTRL_QPN_SHIFT) | + PROGRESS_PARAMS_DS_CNT); + fill_nvmeotcp_progress_params(queue, &wqe->params, seq); +} + +static void +fill_nvmeotcp_static_params(struct mlx5e_nvmeotcp_queue *queue, + struct mlx5_seg_nvmeotcp_static_params *params, + u32 resync_seq, bool zero_copy_en, + bool ddgst_offload_en) +{ + void *ctx = params->ctx; + + MLX5_SET(transport_static_params, ctx, const_1, 1); + MLX5_SET(transport_static_params, ctx, const_2, 2); + MLX5_SET(transport_static_params, ctx, acc_type, + MLX5_TRANSPORT_STATIC_PARAMS_ACC_TYPE_NVMETCP); + MLX5_SET(transport_static_params, ctx, nvme_resync_tcp_sn, resync_seq); + MLX5_SET(transport_static_params, ctx, pda, queue->pda); + MLX5_SET(transport_static_params, ctx, ddgst_en, + queue->dgst & NVME_TCP_DATA_DIGEST_ENABLE); + MLX5_SET(transport_static_params, ctx, ddgst_offload_en, ddgst_offload_en); + MLX5_SET(transport_static_params, ctx, hddgst_en, + queue->dgst & NVME_TCP_HDR_DIGEST_ENABLE); + MLX5_SET(transport_static_params, ctx, hdgst_offload_en, 0); + MLX5_SET(transport_static_params, ctx, ti, + MLX5_TRANSPORT_STATIC_PARAMS_TI_INITIATOR); + MLX5_SET(transport_static_params, ctx, const1, 1); + MLX5_SET(transport_static_params, ctx, zero_copy_en, zero_copy_en); +} + +void +build_nvmeotcp_static_params(struct mlx5e_nvmeotcp_queue *queue, + struct mlx5e_set_nvmeotcp_static_params_wqe *wqe, + u32 resync_seq, bool zerocopy, bool crc_rx) +{ + u8 opc_mod = MLX5_CTRL_SEGMENT_OPC_MOD_UMR_NVMEOTCP_TIR_STATIC_PARAMS; + struct mlx5_wqe_umr_ctrl_seg *ucseg = &wqe->uctrl; + struct mlx5_wqe_ctrl_seg *cseg = &wqe->ctrl; + u32 sqn = queue->sq->icosq.sqn; + u16 pc = queue->sq->icosq.pc; + + memset(wqe, 0, MLX5E_NVMEOTCP_STATIC_PARAMS_WQE_SZ); + + cseg->opmod_idx_opcode = cpu_to_be32((pc << MLX5_WQE_CTRL_WQE_INDEX_SHIFT) | + MLX5_OPCODE_UMR | (opc_mod) << 24); + cseg->qpn_ds = cpu_to_be32((sqn << MLX5_WQE_CTRL_QPN_SHIFT) | + STATIC_PARAMS_DS_CNT); + cseg->imm = cpu_to_be32(queue->tirn << MLX5_WQE_CTRL_TIR_TIS_INDEX_SHIFT); + + ucseg->flags = MLX5_UMR_INLINE; + ucseg->bsf_octowords = + cpu_to_be16(MLX5E_NVMEOTCP_STATIC_PARAMS_OCTWORD_SIZE); + fill_nvmeotcp_static_params(queue, &wqe->params, resync_seq, zerocopy, crc_rx); } static void mlx5e_nvmeotcp_fill_wi(struct mlx5e_nvmeotcp_queue *nvmeotcp_queue, - struct mlx5e_icosq *sq, u32 wqe_bbs, u16 pi) + struct mlx5e_icosq *sq, u32 wqe_bbs, u16 pi, + enum wqe_type type) { struct mlx5e_icosq_wqe_info *wi = &sq->db.wqe_info[pi]; wi->num_wqebbs = wqe_bbs; - wi->wqe_type = MLX5E_ICOSQ_WQE_UMR_NVME_TCP; + switch (type) { + case SET_PSV_UMR: + wi->wqe_type = MLX5E_ICOSQ_WQE_SET_PSV_NVME_TCP; + break; + default: + wi->wqe_type = MLX5E_ICOSQ_WQE_UMR_NVME_TCP; + break; + } + + if (type == SET_PSV_UMR) + wi->nvmeotcp_q.queue = nvmeotcp_queue; +} + +static void +mlx5e_nvmeotcp_rx_post_static_params_wqe(struct mlx5e_nvmeotcp_queue *queue, + u32 resync_seq) +{ + struct mlx5e_set_nvmeotcp_static_params_wqe *wqe; + struct mlx5e_icosq *sq = &queue->sq->icosq; + u16 pi, wqe_bbs; + + spin_lock(&queue->nvmeotcp_icosq_lock); + wqe_bbs = MLX5E_NVMEOTCP_STATIC_PARAMS_WQEBBS; + pi = mlx5e_icosq_get_next_pi(sq, wqe_bbs); + wqe = MLX5E_NVMEOTCP_FETCH_STATIC_PARAMS_WQE(sq, pi); + mlx5e_nvmeotcp_fill_wi(NULL, sq, wqe_bbs, pi, BSF_UMR); + build_nvmeotcp_static_params(queue, wqe, resync_seq, queue->zerocopy, queue->crc_rx); + sq->pc += wqe_bbs; + mlx5e_notify_hw(&sq->wq, sq->pc, sq->uar_map, &wqe->ctrl); + spin_unlock(&queue->nvmeotcp_icosq_lock); +} + +static void +mlx5e_nvmeotcp_rx_post_progress_params_wqe(struct mlx5e_nvmeotcp_queue *queue, + u32 seq) +{ + struct mlx5e_set_nvmeotcp_progress_params_wqe *wqe; + struct mlx5e_icosq *sq = &queue->sq->icosq; + u16 pi, wqe_bbs; + + wqe_bbs = MLX5E_NVMEOTCP_PROGRESS_PARAMS_WQEBBS; + pi = mlx5e_icosq_get_next_pi(sq, wqe_bbs); + wqe = MLX5E_NVMEOTCP_FETCH_PROGRESS_PARAMS_WQE(sq, pi); + mlx5e_nvmeotcp_fill_wi(queue, sq, wqe_bbs, pi, SET_PSV_UMR); + build_nvmeotcp_progress_params(queue, wqe, seq); + sq->pc += wqe_bbs; + mlx5e_notify_hw(&sq->wq, sq->pc, sq->uar_map, &wqe->ctrl); } static void post_klm_wqe(struct mlx5e_nvmeotcp_queue *queue, + enum wqe_type wqe_type, u16 ccid, u32 klm_length, u32 *klm_offset) @@ -102,9 +382,9 @@ post_klm_wqe(struct mlx5e_nvmeotcp_queue *queue, wqe_bbs = DIV_ROUND_UP(wqe_sz, MLX5_SEND_WQE_BB); pi = mlx5e_icosq_get_next_pi(sq, wqe_bbs); wqe = MLX5E_NVMEOTCP_FETCH_KLM_WQE(sq, pi); - mlx5e_nvmeotcp_fill_wi(queue, sq, wqe_bbs, pi); + mlx5e_nvmeotcp_fill_wi(queue, sq, wqe_bbs, pi, wqe_type); build_nvmeotcp_klm_umr(queue, wqe, ccid, cur_klm_entries, *klm_offset, - klm_length); + klm_length, wqe_type); *klm_offset += cur_klm_entries; sq->pc += wqe_bbs; sq->doorbell_cseg = &wqe->ctrl; @@ -112,6 +392,7 @@ post_klm_wqe(struct mlx5e_nvmeotcp_queue *queue, static int mlx5e_nvmeotcp_post_klm_wqe(struct mlx5e_nvmeotcp_queue *queue, + enum wqe_type wqe_type, u16 ccid, u32 klm_length) { @@ -125,35 +406,336 @@ mlx5e_nvmeotcp_post_klm_wqe(struct mlx5e_nvmeotcp_queue *queue, max_wqe_bbs = DIV_ROUND_UP(wqe_sz, MLX5_SEND_WQE_BB); room = mlx5e_stop_room_for_wqe(max_wqe_bbs) * wqes; - if (unlikely(!mlx5e_wqc_has_room_for(&sq->wq, sq->cc, sq->pc, room))) + spin_lock(&queue->nvmeotcp_icosq_lock); + if (unlikely(!mlx5e_wqc_has_room_for(&sq->wq, sq->cc, sq->pc, room))) { + spin_unlock(&queue->nvmeotcp_icosq_lock); return -ENOSPC; + } for (i = 0; i < wqes; i++) - post_klm_wqe(queue, ccid, klm_length, &klm_offset); + post_klm_wqe(queue, wqe_type, ccid, klm_length, &klm_offset); mlx5e_notify_hw(&sq->wq, sq->pc, sq->uar_map, sq->doorbell_cseg); + spin_unlock(&queue->nvmeotcp_icosq_lock); return 0; } +static int mlx5e_create_nvmeotcp_mkey(struct mlx5_core_dev *mdev, + u8 access_mode, + u32 translation_octword_size, + struct mlx5_core_mkey *mkey) +{ + int inlen = MLX5_ST_SZ_BYTES(create_mkey_in); + void *mkc; + u32 *in; + int err; + + in = kvzalloc(inlen, GFP_KERNEL); + if (!in) + return -ENOMEM; + + mkc = MLX5_ADDR_OF(create_mkey_in, in, memory_key_mkey_entry); + MLX5_SET(mkc, mkc, free, 1); + MLX5_SET(mkc, mkc, translations_octword_size, translation_octword_size); + MLX5_SET(mkc, mkc, umr_en, 1); + MLX5_SET(mkc, mkc, lw, 1); + MLX5_SET(mkc, mkc, lr, 1); + MLX5_SET(mkc, mkc, access_mode_1_0, access_mode); + + MLX5_SET(mkc, mkc, qpn, 0xffffff); + MLX5_SET(mkc, mkc, pd, mdev->mlx5e_res.hw_objs.pdn); + + err = mlx5_core_create_mkey(mdev, mkey, in, inlen); + + kvfree(in); + return err; +} + static int mlx5e_nvmeotcp_offload_limits(struct net_device *netdev, struct ulp_ddp_limits *limits) { + struct mlx5e_priv *priv = netdev_priv(netdev); + struct mlx5_core_dev *mdev = priv->mdev; + + limits->max_ddp_sgl_len = mlx5e_get_max_sgl(mdev); return 0; } +static void +mlx5e_nvmeotcp_destroy_sq(struct mlx5e_nvmeotcp_sq *nvmeotcpsq) +{ + mlx5e_deactivate_icosq(&nvmeotcpsq->icosq); + mlx5e_close_icosq(&nvmeotcpsq->icosq); + mlx5e_close_cq(&nvmeotcpsq->icosq.cq); + list_del(&nvmeotcpsq->list); + kfree(nvmeotcpsq); +} + +static int +mlx5e_nvmeotcp_build_icosq(struct mlx5e_nvmeotcp_queue *queue, + struct mlx5e_priv *priv) +{ + u16 max_sgl, max_klm_per_wqe, max_umr_per_ccid, sgl_rest, wqebbs_rest; + struct mlx5e_channel *c = priv->channels.c[queue->channel_ix]; + struct mlx5e_sq_param icosq_param = {0}; + struct dim_cq_moder icocq_moder = {0}; + struct mlx5e_nvmeotcp_sq *nvmeotcp_sq; + struct mlx5e_create_cq_param ccp; + struct mlx5e_icosq *icosq; + int err = -ENOMEM; + u16 log_icosq_sz; + u32 max_wqebbs; + + nvmeotcp_sq = kzalloc(sizeof(*nvmeotcp_sq), GFP_KERNEL); + if (!nvmeotcp_sq) + return err; + + icosq = &nvmeotcp_sq->icosq; + max_sgl = mlx5e_get_max_sgl(priv->mdev); + max_klm_per_wqe = queue->max_klms_per_wqe; + max_umr_per_ccid = max_sgl / max_klm_per_wqe; + sgl_rest = max_sgl % max_klm_per_wqe; + wqebbs_rest = sgl_rest ? MLX5E_KLM_UMR_WQEBBS(sgl_rest) : 0; + max_wqebbs = (MLX5E_KLM_UMR_WQEBBS(max_klm_per_wqe) * + max_umr_per_ccid + wqebbs_rest) * queue->size; + log_icosq_sz = order_base_2(max_wqebbs); + + mlx5e_build_icosq_param(priv->mdev, log_icosq_sz, &icosq_param); + mlx5e_build_create_cq_param(&ccp, c); + err = mlx5e_open_cq(priv, icocq_moder, &icosq_param.cqp, &ccp, &icosq->cq); + if (err) + goto err_nvmeotcp_sq; + + err = mlx5e_open_icosq(c, &priv->channels.params, &icosq_param, icosq); + if (err) + goto close_cq; + + spin_lock_init(&queue->nvmeotcp_icosq_lock); + INIT_LIST_HEAD(&nvmeotcp_sq->list); + spin_lock(&c->nvmeotcp_icosq_lock); + list_add(&nvmeotcp_sq->list, &c->list_nvmeotcpsq); + spin_unlock(&c->nvmeotcp_icosq_lock); + queue->sq = nvmeotcp_sq; + mlx5e_activate_icosq(icosq); + return 0; + +close_cq: + mlx5e_close_cq(&icosq->cq); +err_nvmeotcp_sq: + kfree(nvmeotcp_sq); + + return err; +} + +static void +mlx5e_nvmeotcp_destroy_rx(struct mlx5e_nvmeotcp_queue *queue, + struct mlx5_core_dev *mdev, bool zerocopy) +{ + int i; + + mlx5e_accel_fs_del_sk(queue->fh); + for (i = 0; i < queue->size && zerocopy; i++) + mlx5_core_destroy_mkey(mdev, &queue->ccid_table[i].klm_mkey); + + mlx5e_nvmeotcp_destroy_tir(queue->priv, queue->tirn); + if (zerocopy) { + kfree(queue->ccid_table); + mlx5_destroy_nvmeotcp_tag_buf_table(mdev, queue->tag_buf_table_id); + } + + mlx5e_nvmeotcp_destroy_sq(queue->sq); +} + +static int +mlx5e_nvmeotcp_queue_rx_init(struct mlx5e_nvmeotcp_queue *queue, + struct nvme_tcp_ddp_config *config, + struct net_device *netdev, + bool zerocopy, bool crc) +{ + u8 log_queue_size = order_base_2(config->queue_size); + struct mlx5e_priv *priv = netdev_priv(netdev); + struct mlx5_core_dev *mdev = priv->mdev; + struct sock *sk = queue->sk; + int err, max_sgls, i; + + if (zerocopy) { + if (config->queue_size > + BIT(MLX5_CAP_DEV_NVMEOTCP(mdev, log_max_nvmeotcp_tag_buffer_size))) { + return -EINVAL; + } + + err = mlx5e_create_nvmeotcp_tag_buf_table(mdev, queue, log_queue_size); + if (err) + return err; + } + + err = mlx5e_nvmeotcp_build_icosq(queue, priv); + if (err) + goto destroy_tag_buffer_table; + + /* initializes queue->tirn */ + err = mlx5e_nvmeotcp_create_tir(priv, sk, config, queue, zerocopy, crc); + if (err) + goto destroy_icosq; + + mlx5e_nvmeotcp_rx_post_static_params_wqe(queue, 0); + mlx5e_nvmeotcp_rx_post_progress_params_wqe(queue, tcp_sk(sk)->copied_seq); + + if (zerocopy) { + queue->ccid_table = kcalloc(queue->size, + sizeof(struct nvmeotcp_queue_entry), + GFP_KERNEL); + if (!queue->ccid_table) { + err = -ENOMEM; + goto destroy_tir; + } + + max_sgls = mlx5e_get_max_sgl(mdev); + for (i = 0; i < queue->size; i++) { + err = mlx5e_create_nvmeotcp_mkey(mdev, + MLX5_MKC_ACCESS_MODE_KLMS, + max_sgls, + &queue->ccid_table[i].klm_mkey); + if (err) + goto free_sgl; + } + + err = mlx5e_nvmeotcp_post_klm_wqe(queue, BSF_KLM_UMR, 0, queue->size); + if (err) + goto free_sgl; + } + + if (!(WARN_ON(!wait_for_completion_timeout(&queue->done, msecs_to_jiffies(3000))))) + queue->fh = mlx5e_accel_fs_add_sk(priv, sk, queue->tirn, queue->id); + + if (IS_ERR_OR_NULL(queue->fh)) { + err = -EINVAL; + goto free_sgl; + } + + return 0; + +free_sgl: + while ((i--) && zerocopy) + mlx5_core_destroy_mkey(mdev, &queue->ccid_table[i].klm_mkey); + + if (zerocopy) + kfree(queue->ccid_table); +destroy_tir: + mlx5e_nvmeotcp_destroy_tir(priv, queue->tirn); +destroy_icosq: + mlx5e_nvmeotcp_destroy_sq(queue->sq); +destroy_tag_buffer_table: + if (zerocopy) + mlx5_destroy_nvmeotcp_tag_buf_table(mdev, queue->tag_buf_table_id); + + return err; +} + +#define OCTWORD_SHIFT 4 +#define MAX_DS_VALUE 63 static int mlx5e_nvmeotcp_queue_init(struct net_device *netdev, struct sock *sk, struct ulp_ddp_config *tconfig) { - return 0; + struct nvme_tcp_ddp_config *config = (struct nvme_tcp_ddp_config *)tconfig; + bool crc_rx = ((netdev->features & NETIF_F_HW_ULP_DDP) && + (config->dgst & NVME_TCP_DATA_DIGEST_ENABLE)); + bool zerocopy = (netdev->features & NETIF_F_HW_ULP_DDP); + struct mlx5e_priv *priv = netdev_priv(netdev); + struct mlx5_core_dev *mdev = priv->mdev; + struct mlx5e_nvmeotcp_queue *queue; + int max_wqe_sz_cap, queue_id, err; + + if (tconfig->type != ULP_DDP_NVME) { + err = -EOPNOTSUPP; + goto out; + } + + queue = kzalloc(sizeof(*queue), GFP_KERNEL); + if (!queue) { + err = -ENOMEM; + goto out; + } + + queue_id = ida_simple_get(&priv->nvmeotcp->queue_ids, + MIN_NVMEOTCP_QUEUES, MAX_NVMEOTCP_QUEUES, + GFP_KERNEL); + if (queue_id < 0) { + err = -ENOSPC; + goto free_queue; + } + + queue->crc_rx = crc_rx; + queue->zerocopy = zerocopy; + queue->ulp_ddp_ctx.type = ULP_DDP_NVME; + queue->sk = sk; + queue->id = queue_id; + queue->dgst = config->dgst; + queue->pda = config->cpda; + queue->channel_ix = mlx5e_get_channel_ix_from_io_cpu(priv, + config->io_cpu); + queue->size = config->queue_size; + max_wqe_sz_cap = min_t(int, MAX_DS_VALUE * MLX5_SEND_WQE_DS, + MLX5_CAP_GEN(mdev, max_wqe_sz_sq) << OCTWORD_SHIFT); + queue->max_klms_per_wqe = MLX5E_KLM_ENTRIES_PER_WQE(max_wqe_sz_cap); + queue->priv = priv; + init_completion(&queue->done); + + if (zerocopy || crc_rx) { + err = mlx5e_nvmeotcp_queue_rx_init(queue, config, netdev, + zerocopy, crc_rx); + if (err) + goto remove_queue_id; + } + + err = rhashtable_insert_fast(&priv->nvmeotcp->queue_hash, &queue->hash, + rhash_queues); + if (err) + goto destroy_rx; + + write_lock_bh(&sk->sk_callback_lock); + ulp_ddp_set_ctx(sk, queue); + write_unlock_bh(&sk->sk_callback_lock); + refcount_set(&queue->ref_count, 1); + return err; + +destroy_rx: + if (zerocopy || crc_rx) + mlx5e_nvmeotcp_destroy_rx(queue, mdev, zerocopy); +remove_queue_id: + ida_simple_remove(&priv->nvmeotcp->queue_ids, queue_id); +free_queue: + kfree(queue); +out: + return err; } static void mlx5e_nvmeotcp_queue_teardown(struct net_device *netdev, struct sock *sk) { + struct mlx5e_priv *priv = netdev_priv(netdev); + struct mlx5_core_dev *mdev = priv->mdev; + struct mlx5e_nvmeotcp_queue *queue; + + queue = container_of(ulp_ddp_get_ctx(sk), struct mlx5e_nvmeotcp_queue, ulp_ddp_ctx); + + napi_synchronize(&priv->channels.c[queue->channel_ix]->napi); + + WARN_ON(refcount_read(&queue->ref_count) != 1); + if (queue->zerocopy | queue->crc_rx) + mlx5e_nvmeotcp_destroy_rx(queue, mdev, queue->zerocopy); + + rhashtable_remove_fast(&priv->nvmeotcp->queue_hash, &queue->hash, + rhash_queues); + ida_simple_remove(&priv->nvmeotcp->queue_ids, queue->id); + write_lock_bh(&sk->sk_callback_lock); + ulp_ddp_set_ctx(sk, NULL); + write_unlock_bh(&sk->sk_callback_lock); + mlx5e_nvmeotcp_put_queue(queue); } static int @@ -164,6 +746,16 @@ mlx5e_nvmeotcp_ddp_setup(struct net_device *netdev, return 0; } +void mlx5e_nvmeotcp_ctx_comp(struct mlx5e_icosq_wqe_info *wi) +{ + struct mlx5e_nvmeotcp_queue *queue = wi->nvmeotcp_q.queue; + + if (unlikely(!queue)) + return; + + complete(&queue->done); +} + static int mlx5e_nvmeotcp_ddp_teardown(struct net_device *netdev, struct sock *sk, @@ -188,6 +780,27 @@ static const struct ulp_ddp_dev_ops mlx5e_nvmeotcp_ops = { .ulp_ddp_resync = mlx5e_nvmeotcp_dev_resync, }; +struct mlx5e_nvmeotcp_queue * +mlx5e_nvmeotcp_get_queue(struct mlx5e_nvmeotcp *nvmeotcp, int id) +{ + struct mlx5e_nvmeotcp_queue *queue; + + rcu_read_lock(); + queue = rhashtable_lookup_fast(&nvmeotcp->queue_hash, + &id, rhash_queues); + if (queue && !IS_ERR(queue)) + if (!refcount_inc_not_zero(&queue->ref_count)) + queue = NULL; + rcu_read_unlock(); + return queue; +} + +void mlx5e_nvmeotcp_put_queue(struct mlx5e_nvmeotcp_queue *queue) +{ + if (refcount_dec_and_test(&queue->ref_count)) + kfree(queue); +} + int set_feature_nvme_tcp(struct net_device *netdev, bool enable) { struct mlx5e_priv *priv = netdev_priv(netdev); diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.h b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.h index b4a27a03578e..20141010817d 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.h +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.h @@ -102,6 +102,10 @@ int mlx5e_nvmeotcp_init(struct mlx5e_priv *priv); int set_feature_nvme_tcp(struct net_device *netdev, bool enable); int set_feature_nvme_tcp_crc(struct net_device *netdev, bool enable); void mlx5e_nvmeotcp_cleanup(struct mlx5e_priv *priv); +struct mlx5e_nvmeotcp_queue * +mlx5e_nvmeotcp_get_queue(struct mlx5e_nvmeotcp *nvmeotcp, int id); +void mlx5e_nvmeotcp_put_queue(struct mlx5e_nvmeotcp_queue *queue); +void mlx5e_nvmeotcp_ctx_comp(struct mlx5e_icosq_wqe_info *wi); int mlx5e_nvmeotcp_init_rx(struct mlx5e_priv *priv); void mlx5e_nvmeotcp_cleanup_rx(struct mlx5e_priv *priv); #else diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp_utils.h b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp_utils.h index 329e114d6571..44671e28a9ea 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp_utils.h +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp_utils.h @@ -4,9 +4,77 @@ #define __MLX5E_NVMEOTCP_UTILS_H__ #include "en.h" +#include "en_accel/nvmeotcp.h" + +enum { + MLX5E_NVMEOTCP_PROGRESS_PARAMS_PDU_TRACKER_STATE_START = 0, + MLX5E_NVMEOTCP_PROGRESS_PARAMS_PDU_TRACKER_STATE_TRACKING = 1, + MLX5E_NVMEOTCP_PROGRESS_PARAMS_PDU_TRACKER_STATE_SEARCHING = 2, +}; + +struct mlx5_seg_nvmeotcp_static_params { + u8 ctx[MLX5_ST_SZ_BYTES(transport_static_params)]; +}; + +struct mlx5_seg_nvmeotcp_progress_params { + __be32 tir_num; + u8 ctx[MLX5_ST_SZ_BYTES(nvmeotcp_progress_params)]; +}; + +struct mlx5e_set_nvmeotcp_static_params_wqe { + struct mlx5_wqe_ctrl_seg ctrl; + struct mlx5_wqe_umr_ctrl_seg uctrl; + struct mlx5_mkey_seg mkc; + struct mlx5_seg_nvmeotcp_static_params params; +}; + +struct mlx5e_set_nvmeotcp_progress_params_wqe { + struct mlx5_wqe_ctrl_seg ctrl; + struct mlx5_seg_nvmeotcp_progress_params params; +}; + +struct mlx5e_get_psv_wqe { + struct mlx5_wqe_ctrl_seg ctrl; + struct mlx5_seg_get_psv psv; +}; + +/////////////////////////////////////////// +#define MLX5E_NVMEOTCP_STATIC_PARAMS_WQE_SZ \ + (sizeof(struct mlx5e_set_nvmeotcp_static_params_wqe)) + +#define MLX5E_NVMEOTCP_PROGRESS_PARAMS_WQE_SZ \ + (sizeof(struct mlx5e_set_nvmeotcp_progress_params_wqe)) +#define MLX5E_NVMEOTCP_STATIC_PARAMS_OCTWORD_SIZE \ + (MLX5_ST_SZ_BYTES(transport_static_params) / MLX5_SEND_WQE_DS) + +#define MLX5E_NVMEOTCP_STATIC_PARAMS_WQEBBS \ + (DIV_ROUND_UP(MLX5E_NVMEOTCP_STATIC_PARAMS_WQE_SZ, MLX5_SEND_WQE_BB)) +#define MLX5E_NVMEOTCP_PROGRESS_PARAMS_WQEBBS \ + (DIV_ROUND_UP(MLX5E_NVMEOTCP_PROGRESS_PARAMS_WQE_SZ, MLX5_SEND_WQE_BB)) + +#define MLX5E_NVMEOTCP_FETCH_STATIC_PARAMS_WQE(sq, pi) \ + ((struct mlx5e_set_nvmeotcp_static_params_wqe *)\ + mlx5e_fetch_wqe(&(sq)->wq, pi, sizeof(struct mlx5e_set_nvmeotcp_static_params_wqe))) + +#define MLX5E_NVMEOTCP_FETCH_PROGRESS_PARAMS_WQE(sq, pi) \ + ((struct mlx5e_set_nvmeotcp_progress_params_wqe *)\ + mlx5e_fetch_wqe(&(sq)->wq, pi, sizeof(struct mlx5e_set_nvmeotcp_progress_params_wqe))) #define MLX5E_NVMEOTCP_FETCH_KLM_WQE(sq, pi) \ ((struct mlx5e_umr_wqe *)\ mlx5e_fetch_wqe(&(sq)->wq, pi, sizeof(struct mlx5e_umr_wqe))) +#define MLX5_CTRL_SEGMENT_OPC_MOD_UMR_NVMEOTCP_TIR_PROGRESS_PARAMS 0x4 + +void +build_nvmeotcp_progress_params(struct mlx5e_nvmeotcp_queue *queue, + struct mlx5e_set_nvmeotcp_progress_params_wqe *wqe, + u32 seq); + +void +build_nvmeotcp_static_params(struct mlx5e_nvmeotcp_queue *queue, + struct mlx5e_set_nvmeotcp_static_params_wqe *wqe, + u32 resync_seq, + bool zerocopy, bool crc_rx); + #endif /* __MLX5E_NVMEOTCP_UTILS_H__ */ diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c index 9d821facbca4..bc7b19974ed9 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c @@ -47,6 +47,7 @@ #include "fpga/ipsec.h" #include "en_accel/ipsec_rxtx.h" #include "en_accel/tls_rxtx.h" +#include "en_accel/nvmeotcp.h" #include "en/xdp.h" #include "en/xsk/rx.h" #include "en/health.h" @@ -627,6 +628,9 @@ void mlx5e_free_icosq_descs(struct mlx5e_icosq *sq) #ifdef CONFIG_MLX5_EN_NVMEOTCP case MLX5E_ICOSQ_WQE_UMR_NVME_TCP: break; + case MLX5E_ICOSQ_WQE_SET_PSV_NVME_TCP: + mlx5e_nvmeotcp_ctx_comp(wi); + break; #endif } } @@ -702,6 +706,9 @@ int mlx5e_poll_ico_cq(struct mlx5e_cq *cq) #ifdef CONFIG_MLX5_EN_NVMEOTCP case MLX5E_ICOSQ_WQE_UMR_NVME_TCP: break; + case MLX5E_ICOSQ_WQE_SET_PSV_NVME_TCP: + mlx5e_nvmeotcp_ctx_comp(wi); + break; #endif default: netdev_WARN_ONCE(cq->netdev, From patchwork Thu Jul 22 11:03:05 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Boris Pismenny X-Patchwork-Id: 484318 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-20.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER, INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id EA08CC63793 for ; Thu, 22 Jul 2021 11:05:46 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id CAB4B6121E for ; Thu, 22 Jul 2021 11:05:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231706AbhGVKZK (ORCPT ); Thu, 22 Jul 2021 06:25:10 -0400 Received: from mail-mw2nam10on2089.outbound.protection.outlook.com ([40.107.94.89]:6036 "EHLO NAM10-MW2-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S231792AbhGVKY6 (ORCPT ); Thu, 22 Jul 2021 06:24:58 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=ffVuBqpp+kQF+OnYLMhvZtHfkuxwNIrSN12sgAh9oEhGdOosp22TWC0vpXBlJKEAlW3RKYPYwOYC3Oyjo8qsv8njWz/nI7ebGQQK+p2H0f55/y5GkbxUe4sJe9JojVZrxO5HMXe1+REt317RqQAboLfdkkPfP4Hov21PqiKzR1GaOLpbVGaGjE5SDukQrY98yYdiNQi/VYRYYveOjjLx8YB3u+HO+BpKWcPGOmzNPTL607qBTWTJguK0PGZV9SzMjKfVPBuOd0fNbkd6VAWb0RatMe+cbeOtCrwTLQVpBjlpPP/55gnenW4AumJwD1AuLbQuvY1eNvuAAALAIRus5A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=LMx4bzyWhN3mUnVBY0T21dP6NBiwPhfcJ9AdYIPXP/4=; b=ZRCDLBRRw6cUwaLlr8BNK7Ymv9ewkFOKvT4kXVQ61a3X2uvNI1WJGO6vlr725u8dZWWeLC9yy9uISBNUYUVob3i++vQVSSdMti+B2VzU1mj0ZXP15HdM273HG5XkoBLMvMwfvHAOJhDbbVnQMQNWh+qkfmFkAqp8kuZxDdcs6Rr7WaalWkypIJF2z0u/HjsdBiC+Zco5u3/Hy3PZKzNmAdE3QWmkmgjOZdNgczx5/iRF5QCBb0vfSKtyPSowGaMd1RRbTVS67wBGUgCuxCQQeYsiS2uM/TcDzEmqh1us7HqMswUe1UIBS6+wvbNZEbaSMiEUSBRjg4lpisYsUd0Fwg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.34) smtp.rcpttodomain=marvell.com smtp.mailfrom=nvidia.com; dmarc=pass (p=quarantine sp=none pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=LMx4bzyWhN3mUnVBY0T21dP6NBiwPhfcJ9AdYIPXP/4=; b=LuglwbwmEgPARS68eMulEv+Thu5Dx0TMOmR8fQMRk1JWHt0nUb2uwNS9TqFO3c2XRuQ0pTd16eYVueL1hfOlDQMSaMKQU4qeWhDKUy9QuyvruFRm30K3+vPN6UZ3t5E5w2oEjjyTwDpg5pYBZaFE0Ri+PfqMWixJjw8BJvQd/lmEz8kaQxsY2iJ0iv0l1oUDx9O7gppLADzB1U/Xqr+KqvOPOauhBR1zoAtCnsECyNVSN84WZzPVV2s9tb1wB7f+PWq5CFLCoQneXW3nvq0IinqQAf+ZjDY2SKT6FzkfsvPwjxp0y898IP+UWR3h5FH/yyAPsG8Qt23rTh8724rNJw== Received: from BN9PR03CA0205.namprd03.prod.outlook.com (2603:10b6:408:f9::30) by DM6PR12MB3785.namprd12.prod.outlook.com (2603:10b6:5:1cd::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4331.29; Thu, 22 Jul 2021 11:05:32 +0000 Received: from BN8NAM11FT025.eop-nam11.prod.protection.outlook.com (2603:10b6:408:f9:cafe::4c) by BN9PR03CA0205.outlook.office365.com (2603:10b6:408:f9::30) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4352.25 via Frontend Transport; Thu, 22 Jul 2021 11:05:32 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.34) smtp.mailfrom=nvidia.com; marvell.com; dkim=none (message not signed) header.d=none; marvell.com; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.34 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.34; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.34) by BN8NAM11FT025.mail.protection.outlook.com (10.13.177.136) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4352.24 via Frontend Transport; Thu, 22 Jul 2021 11:05:32 +0000 Received: from HQMAIL107.nvidia.com (172.20.187.13) by HQMAIL107.nvidia.com (172.20.187.13) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Thu, 22 Jul 2021 11:05:31 +0000 Received: from vdi.nvidia.com (172.20.187.5) by mail.nvidia.com (172.20.187.13) with Microsoft SMTP Server id 15.0.1497.2 via Frontend Transport; Thu, 22 Jul 2021 11:05:26 +0000 From: Boris Pismenny To: , , , , , , , , , , CC: , , , , , , Boris Pismenny , Or Gerlitz , Yoray Zack Subject: [PATCH v5 net-next 16/36] net/mlx5e: NVMEoTCP async ddp invalidation Date: Thu, 22 Jul 2021 14:03:05 +0300 Message-ID: <20210722110325.371-17-borisp@nvidia.com> X-Mailer: git-send-email 2.24.1 In-Reply-To: <20210722110325.371-1-borisp@nvidia.com> References: <20210722110325.371-1-borisp@nvidia.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: c9da7b07-05d4-4584-ba1e-08d94d009fb7 X-MS-TrafficTypeDiagnostic: DM6PR12MB3785: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:2399; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 8Vc4xO6iJk2yYwUZebADIJTboKj2XWdS31d3kHyKS9bHZ6pjfD3/If0iQbLcitJ2O15PrchNQeZcsXa6H9ss2trHdID44Rig+MWrpYG3d6QTVAKLRCJ5/GfF2SHeWA7V9ErB7XrBqz0LZcLec8vq0BOHr/1+ZyGyuVLrwWLLz1HZyxnNdH1+N2jnEHxnf1DLl0NJq2O1/e6Lc2F7s6c74JE5xRSlwxhZVGEOi9A3Od7AuLBXh/bz1mv7LzrvM+HtoX45oaxxKk2oyY4O+0xGtJPIP5VDpODrgn39YsEpoSCMdd7gffzMz0vs3BYB06kFWAfImpC9xhmUFJU/ic7PxPl9qr3EI2NAIzxuIQoFbrA3xpv37aSs1OfASMVuPMo41yA0MLfZpWyRzFqNkpayGybA/K93gR+MZ5q9KM0OlIGthY0i5+oirpI0KGaK3t0iBZ0v1AGKeXhtL3zhZBHNhqRAmMAMKyI0Fqv2DIr4WM5ZcPh8NC4/W2GylPuYPmcJQi8SS87MyZ1VYh4DJFn5xvyXex5p3u/LrWxy/b73ulqupwp3Tl5tHi2spIVlAhDZ9DxThEg7NDmYXNZJKSZJ0+DHojxU3DKn35ybn3YyBGv8o0iId+XEYmENcE9KzaH3TlZEKmJj5kGvcRxFF1d7Zwtnj81sbKCKHYPam7M4S3DglqT2TV9E7+fZofvjqXG1nbeGObw4LZIRCftCnG0Ci5zygKMBIt5vnkALxtNkk5c= X-Forefront-Antispam-Report: CIP:216.228.112.34; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:schybrid03.nvidia.com; CAT:NONE; SFS:(4636009)(396003)(136003)(39860400002)(376002)(346002)(36840700001)(46966006)(36756003)(83380400001)(26005)(426003)(54906003)(82740400003)(36906005)(110136005)(1076003)(36860700001)(316002)(8676002)(2906002)(7416002)(478600001)(4326008)(8936002)(82310400003)(86362001)(107886003)(5660300002)(47076005)(6666004)(70586007)(186003)(336012)(7696005)(921005)(2616005)(7636003)(356005)(70206006); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Jul 2021 11:05:32.3033 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: c9da7b07-05d4-4584-ba1e-08d94d009fb7 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.112.34]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BN8NAM11FT025.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR12MB3785 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Ben Ben-Ishay Teardown ddp contexts asynchronously by posting a WQE, and calling back to nvme-tcp when the corresponding CQE is received. Signed-off-by: Boris Pismenny Signed-off-by: Ben Ben-Ishay Signed-off-by: Or Gerlitz Signed-off-by: Yoray Zack --- .../net/ethernet/mellanox/mlx5/core/en/txrx.h | 4 ++ .../mellanox/mlx5/core/en_accel/nvmeotcp.c | 66 ++++++++++++++++--- .../mellanox/mlx5/core/en_accel/nvmeotcp.h | 1 + .../net/ethernet/mellanox/mlx5/core/en_rx.c | 6 ++ 4 files changed, 69 insertions(+), 8 deletions(-) diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h b/drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h index 497c49f28d8a..f0190ee6e42c 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h +++ b/drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h @@ -43,6 +43,7 @@ enum mlx5e_icosq_wqe_type { #endif #ifdef CONFIG_MLX5_EN_NVMEOTCP MLX5E_ICOSQ_WQE_UMR_NVME_TCP, + MLX5E_ICOSQ_WQE_UMR_NVME_TCP_INVALIDATE, MLX5E_ICOSQ_WQE_SET_PSV_NVME_TCP, #endif }; @@ -191,6 +192,9 @@ struct mlx5e_icosq_wqe_info { struct { struct mlx5e_nvmeotcp_queue *queue; } nvmeotcp_q; + struct { + struct nvmeotcp_queue_entry *entry; + } nvmeotcp_qe; #endif }; }; diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c index 2283b2a799f8..864b080cc8fc 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c @@ -152,6 +152,7 @@ enum wqe_type { BSF_KLM_UMR = 1, SET_PSV_UMR = 2, BSF_UMR = 3, + KLM_INV_UMR = 4, }; static void @@ -208,6 +209,13 @@ build_nvmeotcp_klm_umr(struct mlx5e_nvmeotcp_queue *queue, MLX5E_KLM_UMR_DS_CNT(ALIGN(klm_entries, KLM_ALIGNMENT))); cseg->general_id = cpu_to_be32(id); + if (!klm_entries) { /* this is invalidate */ + ucseg->mkey_mask = cpu_to_be64(MLX5_MKEY_MASK_FREE); + ucseg->flags = MLX5_UMR_INLINE; + mkc->status = MLX5_MKEY_STATUS_FREE; + return; + } + if (klm_type == KLM_UMR && !klm_offset) { ucseg->mkey_mask |= cpu_to_be64(MLX5_MKEY_MASK_XLT_OCT_SIZE | MLX5_MKEY_MASK_LEN | MLX5_MKEY_MASK_FREE); @@ -309,8 +317,8 @@ build_nvmeotcp_static_params(struct mlx5e_nvmeotcp_queue *queue, static void mlx5e_nvmeotcp_fill_wi(struct mlx5e_nvmeotcp_queue *nvmeotcp_queue, - struct mlx5e_icosq *sq, u32 wqe_bbs, u16 pi, - enum wqe_type type) + struct mlx5e_icosq *sq, u32 wqe_bbs, + u16 pi, u16 ccid, enum wqe_type type) { struct mlx5e_icosq_wqe_info *wi = &sq->db.wqe_info[pi]; @@ -319,12 +327,17 @@ mlx5e_nvmeotcp_fill_wi(struct mlx5e_nvmeotcp_queue *nvmeotcp_queue, case SET_PSV_UMR: wi->wqe_type = MLX5E_ICOSQ_WQE_SET_PSV_NVME_TCP; break; + case KLM_INV_UMR: + wi->wqe_type = MLX5E_ICOSQ_WQE_UMR_NVME_TCP_INVALIDATE; + break; default: wi->wqe_type = MLX5E_ICOSQ_WQE_UMR_NVME_TCP; break; } - if (type == SET_PSV_UMR) + if (type == KLM_INV_UMR) + wi->nvmeotcp_qe.entry = &nvmeotcp_queue->ccid_table[ccid]; + else if (type == SET_PSV_UMR) wi->nvmeotcp_q.queue = nvmeotcp_queue; } @@ -340,7 +353,7 @@ mlx5e_nvmeotcp_rx_post_static_params_wqe(struct mlx5e_nvmeotcp_queue *queue, wqe_bbs = MLX5E_NVMEOTCP_STATIC_PARAMS_WQEBBS; pi = mlx5e_icosq_get_next_pi(sq, wqe_bbs); wqe = MLX5E_NVMEOTCP_FETCH_STATIC_PARAMS_WQE(sq, pi); - mlx5e_nvmeotcp_fill_wi(NULL, sq, wqe_bbs, pi, BSF_UMR); + mlx5e_nvmeotcp_fill_wi(NULL, sq, wqe_bbs, pi, 0, BSF_UMR); build_nvmeotcp_static_params(queue, wqe, resync_seq, queue->zerocopy, queue->crc_rx); sq->pc += wqe_bbs; mlx5e_notify_hw(&sq->wq, sq->pc, sq->uar_map, &wqe->ctrl); @@ -358,7 +371,7 @@ mlx5e_nvmeotcp_rx_post_progress_params_wqe(struct mlx5e_nvmeotcp_queue *queue, wqe_bbs = MLX5E_NVMEOTCP_PROGRESS_PARAMS_WQEBBS; pi = mlx5e_icosq_get_next_pi(sq, wqe_bbs); wqe = MLX5E_NVMEOTCP_FETCH_PROGRESS_PARAMS_WQE(sq, pi); - mlx5e_nvmeotcp_fill_wi(queue, sq, wqe_bbs, pi, SET_PSV_UMR); + mlx5e_nvmeotcp_fill_wi(queue, sq, wqe_bbs, pi, 0, SET_PSV_UMR); build_nvmeotcp_progress_params(queue, wqe, seq); sq->pc += wqe_bbs; mlx5e_notify_hw(&sq->wq, sq->pc, sq->uar_map, &wqe->ctrl); @@ -382,7 +395,8 @@ post_klm_wqe(struct mlx5e_nvmeotcp_queue *queue, wqe_bbs = DIV_ROUND_UP(wqe_sz, MLX5_SEND_WQE_BB); pi = mlx5e_icosq_get_next_pi(sq, wqe_bbs); wqe = MLX5E_NVMEOTCP_FETCH_KLM_WQE(sq, pi); - mlx5e_nvmeotcp_fill_wi(queue, sq, wqe_bbs, pi, wqe_type); + mlx5e_nvmeotcp_fill_wi(queue, sq, wqe_bbs, pi, ccid, + klm_length ? KLM_UMR : KLM_INV_UMR); build_nvmeotcp_klm_umr(queue, wqe, ccid, cur_klm_entries, *klm_offset, klm_length, wqe_type); *klm_offset += cur_klm_entries; @@ -400,8 +414,13 @@ mlx5e_nvmeotcp_post_klm_wqe(struct mlx5e_nvmeotcp_queue *queue, struct mlx5e_icosq *sq = &queue->sq->icosq; /* TODO: set stricter wqe_sz; using max for now */ - wqes = DIV_ROUND_UP(klm_length, queue->max_klms_per_wqe); - wqe_sz = MLX5E_KLM_UMR_WQE_SZ(queue->max_klms_per_wqe); + if (klm_length == 0) { + wqes = 1; + wqe_sz = MLX5E_NVMEOTCP_STATIC_PARAMS_WQEBBS; + } else { + wqes = DIV_ROUND_UP(klm_length, queue->max_klms_per_wqe); + wqe_sz = MLX5E_KLM_UMR_WQE_SZ(queue->max_klms_per_wqe); + } max_wqe_bbs = DIV_ROUND_UP(wqe_sz, MLX5_SEND_WQE_BB); @@ -746,6 +765,24 @@ mlx5e_nvmeotcp_ddp_setup(struct net_device *netdev, return 0; } +void mlx5e_nvmeotcp_ddp_inv_done(struct mlx5e_icosq_wqe_info *wi) +{ + struct nvmeotcp_queue_entry *q_entry = wi->nvmeotcp_qe.entry; + struct mlx5e_nvmeotcp_queue *queue = q_entry->queue; + struct mlx5_core_dev *mdev = queue->priv->mdev; + struct ulp_ddp_io *ddp = q_entry->ddp; + const struct ulp_ddp_ulp_ops *ulp_ops; + + dma_unmap_sg(mdev->device, ddp->sg_table.sgl, + q_entry->sgl_length, DMA_FROM_DEVICE); + + q_entry->sgl_length = 0; + + ulp_ops = inet_csk(queue->sk)->icsk_ulp_ddp_ops; + if (ulp_ops && ulp_ops->ddp_teardown_done) + ulp_ops->ddp_teardown_done(q_entry->ddp_ctx); +} + void mlx5e_nvmeotcp_ctx_comp(struct mlx5e_icosq_wqe_info *wi) { struct mlx5e_nvmeotcp_queue *queue = wi->nvmeotcp_q.queue; @@ -762,6 +799,19 @@ mlx5e_nvmeotcp_ddp_teardown(struct net_device *netdev, struct ulp_ddp_io *ddp, void *ddp_ctx) { + struct mlx5e_nvmeotcp_queue *queue; + struct mlx5e_priv *priv = netdev_priv(netdev); + struct nvmeotcp_queue_entry *q_entry; + + queue = container_of(ulp_ddp_get_ctx(sk), struct mlx5e_nvmeotcp_queue, ulp_ddp_ctx); + q_entry = &queue->ccid_table[ddp->command_id]; + WARN_ON(q_entry->sgl_length == 0); + + q_entry->ddp_ctx = ddp_ctx; + q_entry->queue = queue; + + mlx5e_nvmeotcp_post_klm_wqe(queue, KLM_UMR, ddp->command_id, 0); + return 0; } diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.h b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.h index 20141010817d..b9642e130b97 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.h +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.h @@ -105,6 +105,7 @@ void mlx5e_nvmeotcp_cleanup(struct mlx5e_priv *priv); struct mlx5e_nvmeotcp_queue * mlx5e_nvmeotcp_get_queue(struct mlx5e_nvmeotcp *nvmeotcp, int id); void mlx5e_nvmeotcp_put_queue(struct mlx5e_nvmeotcp_queue *queue); +void mlx5e_nvmeotcp_ddp_inv_done(struct mlx5e_icosq_wqe_info *wi); void mlx5e_nvmeotcp_ctx_comp(struct mlx5e_icosq_wqe_info *wi); int mlx5e_nvmeotcp_init_rx(struct mlx5e_priv *priv); void mlx5e_nvmeotcp_cleanup_rx(struct mlx5e_priv *priv); diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c index bc7b19974ed9..e92dd4666955 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c @@ -628,6 +628,9 @@ void mlx5e_free_icosq_descs(struct mlx5e_icosq *sq) #ifdef CONFIG_MLX5_EN_NVMEOTCP case MLX5E_ICOSQ_WQE_UMR_NVME_TCP: break; + case MLX5E_ICOSQ_WQE_UMR_NVME_TCP_INVALIDATE: + mlx5e_nvmeotcp_ddp_inv_done(wi); + break; case MLX5E_ICOSQ_WQE_SET_PSV_NVME_TCP: mlx5e_nvmeotcp_ctx_comp(wi); break; @@ -706,6 +709,9 @@ int mlx5e_poll_ico_cq(struct mlx5e_cq *cq) #ifdef CONFIG_MLX5_EN_NVMEOTCP case MLX5E_ICOSQ_WQE_UMR_NVME_TCP: break; + case MLX5E_ICOSQ_WQE_UMR_NVME_TCP_INVALIDATE: + mlx5e_nvmeotcp_ddp_inv_done(wi); + break; case MLX5E_ICOSQ_WQE_SET_PSV_NVME_TCP: mlx5e_nvmeotcp_ctx_comp(wi); break; From patchwork Thu Jul 22 11:03:06 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Boris Pismenny X-Patchwork-Id: 485104 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-20.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER, INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2292DC6377D for ; Thu, 22 Jul 2021 11:05:50 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 0B0336121E for ; Thu, 22 Jul 2021 11:05:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231783AbhGVKZN (ORCPT ); Thu, 22 Jul 2021 06:25:13 -0400 Received: from mail-mw2nam12on2072.outbound.protection.outlook.com ([40.107.244.72]:1217 "EHLO NAM12-MW2-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S231721AbhGVKZC (ORCPT ); Thu, 22 Jul 2021 06:25:02 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=DWKBVB2AGCp3LtyjLBZcGHyNF2Ohojz/nrzpAfueO0QOe1wiN40AWAU4F1YyT4Y4yyJ5FF3Tyk8fLi3vnKX9pUbYqC0HRq1fXlYjZQRrDpWsrlmixPWVA526jXzyoslEsaFYsyEHD1Bt3dWEm9M15wGcMcIOd5nLERtL4zapoT2YhFnyHd5YX5mizMiFqLf/Ct41kVm2+vya1BYr4aiGNqsoCHSh6fFcc1KU6oRZeLQeeNirDjZP11oIWDLC6dHa8/mnTxqmSg6+7rQGP3yNSKPB1SIJbyh6/aU8V8OyydNbEF0RPErw3qw07C4LSWfkHzCNfxS7nbiM+1g1iorG8Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=MSghkhcHtwYVrIl5SlfHjm8XRASFNZLr9jSOYysXWmI=; b=MJKpYV7I7ah6PPg2ZEA6SS4U3dbv55jpiHux38ldq1MLWq57iJ4DNnkKynmAywpNJlhubuvtF0/ySr0960Elj1UxghQwGuDzPHfAcjFtYmSIkLqxxWljJ0ndY5AJabHwrnDp0UnvdRkInrsd2OsNeWcNfTTGHgoMX7NaYUInNwcQEZe3iNU1j2Qzn6Ll48MtL92UkhQvUUaPhl6dHMcUu1yRDLgR7tFDR9/XuKqNUbGuYMV8YdeWPA4M9YM1Rq457gsn7gYhtgxjvk3z3PFOIu9MdWd+IwD2qZAd2Uz94W2anHPsUyjacYHyUQMz+Qd1PZf+tm9bEt/2cLjOHqa+fA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.32) smtp.rcpttodomain=marvell.com smtp.mailfrom=nvidia.com; dmarc=pass (p=quarantine sp=none pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=MSghkhcHtwYVrIl5SlfHjm8XRASFNZLr9jSOYysXWmI=; b=Nz3VqiYALA4eqTu0zBGDW4MFb05MEWj004C+LVIxI/LhqbTiVza1pzmTAiWgG8mGwW3zfQ0NtcuQ267tdLYeyC2Gqv+8+TScvnZ2b27+FXW7DldFbccMt/sTWXurYzoG6BwEjG+6vIpc6CAGOXrh3xU9lkx8hmy538nAnOP1jhKeuqIep9bgiqA6rJ7hb/3JVP3LY6298HVhUGybRBzSkMmx8wBQce+cDVun2JRATHeI4ym0X8x3e6kma9aW6Lz9QhO4NXd6oGCVyPX1o4Rl+E5zRhhIYrDJhQTfti2n0M6++W9+R+Z5ZWEPqS2RbR+fZ/oSXOl4T9b3MQdTR9JXMg== Received: from DM3PR14CA0144.namprd14.prod.outlook.com (2603:10b6:0:53::28) by SN1PR12MB2384.namprd12.prod.outlook.com (2603:10b6:802:25::25) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4331.23; Thu, 22 Jul 2021 11:05:36 +0000 Received: from DM6NAM11FT041.eop-nam11.prod.protection.outlook.com (2603:10b6:0:53:cafe::a7) by DM3PR14CA0144.outlook.office365.com (2603:10b6:0:53::28) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4352.25 via Frontend Transport; Thu, 22 Jul 2021 11:05:36 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.32) smtp.mailfrom=nvidia.com; marvell.com; dkim=none (message not signed) header.d=none; marvell.com; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.32 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.32; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.32) by DM6NAM11FT041.mail.protection.outlook.com (10.13.172.98) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4352.24 via Frontend Transport; Thu, 22 Jul 2021 11:05:36 +0000 Received: from HQMAIL107.nvidia.com (172.20.187.13) by HQMAIL109.nvidia.com (172.20.187.15) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Thu, 22 Jul 2021 04:05:35 -0700 Received: from vdi.nvidia.com (172.20.187.5) by mail.nvidia.com (172.20.187.13) with Microsoft SMTP Server id 15.0.1497.2 via Frontend Transport; Thu, 22 Jul 2021 11:05:31 +0000 From: Boris Pismenny To: , , , , , , , , , , CC: , , , , , , Boris Pismenny , Or Gerlitz , Yoray Zack Subject: [PATCH v5 net-next 17/36] net/mlx5e: NVMEoTCP ddp setup and resync Date: Thu, 22 Jul 2021 14:03:06 +0300 Message-ID: <20210722110325.371-18-borisp@nvidia.com> X-Mailer: git-send-email 2.24.1 In-Reply-To: <20210722110325.371-1-borisp@nvidia.com> References: <20210722110325.371-1-borisp@nvidia.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: f1776bb6-5ad9-4234-d866-08d94d00a218 X-MS-TrafficTypeDiagnostic: SN1PR12MB2384: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:3513; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: EwkAvsDtpIvDRLKdnxIshKY8Zb6OpJQJQ4yQEcoS13rt+AeRrT3FysfUnQbYKEMzi5RC5YJLTOqWRN0TEufCocVO1hh5pyIIlxEpV1SR7KAH4U/8fyelGw99IP8KK5lQxFX8x20ECcJFExbxqdwp/RfVpn2C10Vl8O4nvcH1JAh2hqQliK0d4nw8WNLHOI5ndjCmmRbfxDb9IseyW0Fdbi+0XbGWDx+9qQj6fshuqTBccY50XKlRxeGYCYf1/vJ1C0rXWzMHMNteXThLvoeK+ttuYQA+tC5BM/ROrVFD1czYZOPSTmFMy+hwf/y9REWInLo6LemmKo2HBsdBcnBl2vLSOo1XlgovO9nDjsTNkvHzqkXoMuDwtoL2NBlFeVbk8aNO5/JBlIxW0NqS9cDWNoT5K8aqr90NTVP2NgPl3pyFmSxuqo7USs+kmdm58f76xk+roUOt47K1pjgZvI/W3S/ts+pMZvwXbsj5Rk1+F22gBLNeS0mPHw4N0Svh/kzqt9ePwmdr+vtl6wXJPKK8uJLoeXg+4W+EheOVdR/QtR0duDRc5H9bNt9Ij8axWesZAX7usU75AqBU9JPnaM+WSVhBSz3RSuLaxPSFB46veELmSBcJkE3vp6p8y2/80fR395RaxT1aFT4x/E22ekEemg9EUlqAjVJN9Lwj6cQfOraHsvUv+0qjPNhAFSmABhZCJGITk5MyDCqJr5nPpUBRUiOYDutY4gsSZUoWx0JE1fk= X-Forefront-Antispam-Report: CIP:216.228.112.32; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:schybrid01.nvidia.com; CAT:NONE; SFS:(4636009)(46966006)(36840700001)(5660300002)(82310400003)(7416002)(8676002)(47076005)(2906002)(6666004)(8936002)(107886003)(36860700001)(54906003)(7696005)(4326008)(83380400001)(26005)(86362001)(336012)(356005)(316002)(36756003)(186003)(426003)(508600001)(110136005)(7636003)(1076003)(70206006)(70586007)(921005)(2616005); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Jul 2021 11:05:36.3502 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: f1776bb6-5ad9-4234-d866-08d94d00a218 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.112.32]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DM6NAM11FT041.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN1PR12MB2384 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Ben Ben-Ishay NVMEoTCP offload uses buffer registration for every NVME request to perform direct data placement, The registration is done via KLM UMR WQE's. The driver resync handler advertise the software resync response via static params WQE. Signed-off-by: Boris Pismenny Signed-off-by: Ben Ben-Ishay Signed-off-by: Or Gerlitz Signed-off-by: Yoray Zack --- .../mellanox/mlx5/core/en_accel/nvmeotcp.c | 29 +++++++++++++++++++ 1 file changed, 29 insertions(+) diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c index 864b080cc8fc..4fdfbe4468ee 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c @@ -762,6 +762,30 @@ mlx5e_nvmeotcp_ddp_setup(struct net_device *netdev, struct sock *sk, struct ulp_ddp_io *ddp) { + struct mlx5e_priv *priv = netdev_priv(netdev); + struct scatterlist *sg = ddp->sg_table.sgl; + struct mlx5e_nvmeotcp_queue *queue; + struct mlx5_core_dev *mdev; + int i, size = 0, count = 0; + + queue = container_of(ulp_ddp_get_ctx(sk), struct mlx5e_nvmeotcp_queue, ulp_ddp_ctx); + + mdev = queue->priv->mdev; + count = dma_map_sg(mdev->device, ddp->sg_table.sgl, ddp->nents, + DMA_FROM_DEVICE); + + if (WARN_ON(count > mlx5e_get_max_sgl(mdev))) + return -ENOSPC; + + for (i = 0; i < count; i++) + size += sg[i].length; + + queue->ccid_table[ddp->command_id].size = size; + queue->ccid_table[ddp->command_id].ddp = ddp; + queue->ccid_table[ddp->command_id].sgl = sg; + queue->ccid_table[ddp->command_id].ccid_gen++; + queue->ccid_table[ddp->command_id].sgl_length = count; + return 0; } @@ -819,6 +843,11 @@ static void mlx5e_nvmeotcp_dev_resync(struct net_device *netdev, struct sock *sk, u32 seq) { + struct mlx5e_nvmeotcp_queue *queue = + container_of(ulp_ddp_get_ctx(sk), struct mlx5e_nvmeotcp_queue, ulp_ddp_ctx); + + queue->after_resync_cqe = 1; + mlx5e_nvmeotcp_rx_post_static_params_wqe(queue, seq); } static const struct ulp_ddp_dev_ops mlx5e_nvmeotcp_ops = { From patchwork Thu Jul 22 11:03:07 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Boris Pismenny X-Patchwork-Id: 484317 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-20.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER, INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 37CEDC63797 for ; Thu, 22 Jul 2021 11:05:54 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 219FB6120D for ; Thu, 22 Jul 2021 11:05:54 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231708AbhGVKZR (ORCPT ); Thu, 22 Jul 2021 06:25:17 -0400 Received: from mail-mw2nam10on2076.outbound.protection.outlook.com ([40.107.94.76]:3360 "EHLO NAM10-MW2-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S231635AbhGVKZI (ORCPT ); Thu, 22 Jul 2021 06:25:08 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=QilGpbRlZOWCTrjR9FdVfcemeJfEDi4wPgKARnApjHZKlW66bqxI5+HNDhZ/dWp0b7veS6TrHa+IJos126DSEKz59LkMAz4+eehjmKbQP/Itb/+3nShpIcBn/JuV+Yh4SHClaQWt6TqussIRrxMcjNdfheoWJV7l5qV1COCTgXwdXAlJZ6HPrHOLFpGaHQ/QnqkfHv6lNY/VCWADQrWFLJvKZyL45veeuiS32iWO0vLJoERNgyDbIX4LQhV7PHk31rMLlIv/q/Ux5B6JOKZKcda5ft5CfCv0iqXZXPdClb4nDL/63VTYDY3HSXxeKNE98otwHOu4aMljLQbvjbneNw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=ih0DogM3eCmXM6lcaCWNE5D6+pFQiv7HQwXNqPd2cjE=; b=iO9UHkufITfBZOGSHPfDqj51s/+c+xdV+W6QXtlF98phU2iipNDLrh5R3e01IAFmFNm7OStLIvJo/FVlXNjibh4zB0eu7cRgwmSrKmRkU2zSVKkyzBOPYpvbSZuw4ZTZ5lfmf88N4zArqYcWzUTn2MWzvM9+WbnJ38DQGJChCQuDB2HK1iDWQo/tcUpuqFiMHMxT4dDWvzvtwHHYJnTQe0qLPv01hHsWOMWE5AyOj0U87ZLaJrXlsVhSrHioL7egyVJAoAqWVDAAHuMmWSNpW4NC5NVXLInq7IsP0VpIp6eaAz2KXt60zynnk4jAq2lUHUtL0D1p4IU4z/dJCYVXJg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.35) smtp.rcpttodomain=marvell.com smtp.mailfrom=nvidia.com; dmarc=pass (p=quarantine sp=none pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=ih0DogM3eCmXM6lcaCWNE5D6+pFQiv7HQwXNqPd2cjE=; b=hsycMXJysGKmLm9e/BStdoLm2yKWEIMVl4aQmDVMtB+etR0MEYjzmNQxkDHaj87+luta+O8IQSTBWusImvRtfFSlC+PctYs1IUFi8A8jpifvLziPwFSicDACelr/bn5C+Q9WaMCROMKugpW1sdJ0TVzddEaXpsdMCfjobvBK/9CVhAxfwLNlTRtMV9u1wCbxqHFEJJnBEnDun0vH7W/LJwxG7t6KJPD6AllJFcGHK9i88UqAEHnsjmzG+Mt9pg/S01GRyO6bczDKiYN84N9nlrjRjndtKHLuvVrNVj6W4Wza8jF8lZRSy7m27ecwupM5ULS9mTv6sBg6EUGGwrOJkA== Received: from DM6PR02CA0089.namprd02.prod.outlook.com (2603:10b6:5:1f4::30) by BN8PR12MB3076.namprd12.prod.outlook.com (2603:10b6:408:61::27) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4331.23; Thu, 22 Jul 2021 11:05:41 +0000 Received: from DM6NAM11FT022.eop-nam11.prod.protection.outlook.com (2603:10b6:5:1f4:cafe::4b) by DM6PR02CA0089.outlook.office365.com (2603:10b6:5:1f4::30) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4352.26 via Frontend Transport; Thu, 22 Jul 2021 11:05:41 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.35) smtp.mailfrom=nvidia.com; marvell.com; dkim=none (message not signed) header.d=none; marvell.com; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.35 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.35; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.35) by DM6NAM11FT022.mail.protection.outlook.com (10.13.172.210) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4352.24 via Frontend Transport; Thu, 22 Jul 2021 11:05:41 +0000 Received: from HQMAIL107.nvidia.com (172.20.187.13) by HQMAIL111.nvidia.com (172.20.187.18) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Thu, 22 Jul 2021 11:05:40 +0000 Received: from vdi.nvidia.com (172.20.187.5) by mail.nvidia.com (172.20.187.13) with Microsoft SMTP Server id 15.0.1497.2 via Frontend Transport; Thu, 22 Jul 2021 11:05:36 +0000 From: Boris Pismenny To: , , , , , , , , , , CC: , , , , , , Boris Pismenny , Or Gerlitz , Yoray Zack Subject: [PATCH v5 net-next 18/36] net/mlx5e: NVMEoTCP, data-path for DDP+DDGST offload Date: Thu, 22 Jul 2021 14:03:07 +0300 Message-ID: <20210722110325.371-19-borisp@nvidia.com> X-Mailer: git-send-email 2.24.1 In-Reply-To: <20210722110325.371-1-borisp@nvidia.com> References: <20210722110325.371-1-borisp@nvidia.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: ed132b1b-499f-4575-2072-08d94d00a506 X-MS-TrafficTypeDiagnostic: BN8PR12MB3076: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:10000; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: UuNaxI+Pp31kFm3MUvEOBEXaINJOfKe6VlnvW957DF5dJUG0sPqXzfSNx1Rx7JEXkq5JbujJ9kuB/F24d07sEHo9e3SmH4U4DgicafEFPuusBlNc9YGBqp3/ASIRi0WG74bEQtijtrUZbhN/Twv+vXB8bsDKRe8CkImM2SMhUb1+hG3tw2Ymzlq1Mo41/bfH8X1Pp1o6lTHvItLDTSvf0WvBnX+XmzHXdssiuOQ3DjPZz0fDLzoDWMiLUJKLHwj5W3th6H7t1VucmvR2DgjJxkeYy6+nwdHMi9mG/fAhWJlIsXDp8A/9aNhWSlD7oNlpDDXEJq+DYarisSe2IZI8xlbHGVUE1sgVqE1MmGvKA/u7jYqKy/dSB68hAatOih+7dklDdnx0w2b8prqr3pvuPUvYzhaZ+pR29S2NJaYwY2eDUXBMqvurEt3bhu53jlDN+6rKaIR5F+Hhiyc/wKDw8GGjXxFR+wh3jyNR71CciluuhCbO63Yb8gsbJAkYYLAINUEHB8RJkH21uzr5NEy1RV/Wds5BgGUCjNw4kKN5P7Sk4jMh4KPGJ6f7LhXQHW8gB1LIAG06ElTZHdQTCWl7OCPq6NPrBLNHTQiYHlVc+6Jwk82K0qcOXKZ77qt5qkl3gAEnGT9TVz6MKSKKjUJPP694XOYqy/nFbzIhYJOD90NMMwLevgWpOIJPWyiS1yHBeoHR44j5qvAMPYM7IAieBFCk7pYqFVmzGT09aOTSCXZSmvl3EAuymUwif9sejGlX X-Forefront-Antispam-Report: CIP:216.228.112.35; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:schybrid04.nvidia.com; CAT:NONE; SFS:(4636009)(39860400002)(376002)(396003)(136003)(346002)(46966006)(36840700001)(83380400001)(36906005)(8936002)(5660300002)(6666004)(2906002)(186003)(356005)(7636003)(426003)(8676002)(2616005)(30864003)(47076005)(7696005)(7416002)(54906003)(86362001)(26005)(70206006)(336012)(478600001)(1076003)(4326008)(36756003)(82740400003)(110136005)(316002)(107886003)(921005)(70586007)(36860700001)(82310400003)(21314003); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Jul 2021 11:05:41.2537 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: ed132b1b-499f-4575-2072-08d94d00a506 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.112.35]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DM6NAM11FT022.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN8PR12MB3076 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Ben Ben-Ishay This patch implements the data-path for direct data placement (DDP) and DDGST offloads. NVMEoTCP DDP constructs an SKB from each CQE, while pointing at NVME destination buffers. In turn, this enables the offload, as the NVMe-TCP layer will skip the copy when src == dst. Additionally, this patch adds support for DDGST (CRC32) offload. HW will report DDGST offload only if it has not encountered an error in the received packet. We pass this indication in skb->ddp_crc up the stack to NVMe-TCP to skip computing the DDGST if all corresponding SKBs were verified by HW. This patch also handles context resynchronization requests made by NIC HW. The resync request is passed to the NVMe-TCP layer to be handled at a later point in time. Finally, we also use the skb->ddp_crc bit to avoid skb_condense. This is critical as every SKB that uses DDP has a hole that fits perfectly with skb_condense's policy, but filling this hole is counter-productive as the data there already resides in its destination buffer. Signed-off-by: Boris Pismenny Signed-off-by: Ben Ben-Ishay Signed-off-by: Or Gerlitz Signed-off-by: Yoray Zack --- .../net/ethernet/mellanox/mlx5/core/Makefile | 2 +- drivers/net/ethernet/mellanox/mlx5/core/en.h | 1 + .../ethernet/mellanox/mlx5/core/en/xsk/rx.c | 1 + .../ethernet/mellanox/mlx5/core/en/xsk/rx.h | 1 + .../mlx5/core/en_accel/nvmeotcp_rxtx.c | 248 ++++++++++++++++++ .../mlx5/core/en_accel/nvmeotcp_rxtx.h | 43 +++ .../net/ethernet/mellanox/mlx5/core/en_rx.c | 41 ++- 7 files changed, 330 insertions(+), 7 deletions(-) create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp_rxtx.c create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp_rxtx.h diff --git a/drivers/net/ethernet/mellanox/mlx5/core/Makefile b/drivers/net/ethernet/mellanox/mlx5/core/Makefile index 0ae9e5e38ec7..992b396907ae 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/Makefile +++ b/drivers/net/ethernet/mellanox/mlx5/core/Makefile @@ -102,4 +102,4 @@ mlx5_core-$(CONFIG_MLX5_SF) += sf/vhca_event.o sf/dev/dev.o sf/dev/driver.o # mlx5_core-$(CONFIG_MLX5_SF_MANAGER) += sf/cmd.o sf/hw_table.o sf/devlink.o -mlx5_core-$(CONFIG_MLX5_EN_NVMEOTCP) += en_accel/fs_tcp.o en_accel/nvmeotcp.o +mlx5_core-$(CONFIG_MLX5_EN_NVMEOTCP) += en_accel/fs_tcp.o en_accel/nvmeotcp.o en_accel/nvmeotcp_rxtx.o diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en.h b/drivers/net/ethernet/mellanox/mlx5/core/en.h index d88ecd04d5da..c87f32492ea4 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en.h +++ b/drivers/net/ethernet/mellanox/mlx5/core/en.h @@ -589,6 +589,7 @@ struct mlx5e_rq; typedef void (*mlx5e_fp_handle_rx_cqe)(struct mlx5e_rq*, struct mlx5_cqe64*); typedef struct sk_buff * (*mlx5e_fp_skb_from_cqe_mpwrq)(struct mlx5e_rq *rq, struct mlx5e_mpw_info *wi, + struct mlx5_cqe64 *cqe, u16 cqe_bcnt, u32 head_offset, u32 page_idx); typedef struct sk_buff * (*mlx5e_fp_skb_from_cqe)(struct mlx5e_rq *rq, struct mlx5_cqe64 *cqe, diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/xsk/rx.c b/drivers/net/ethernet/mellanox/mlx5/core/en/xsk/rx.c index 8e7b877d8a12..9a6fbd1b1c34 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en/xsk/rx.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en/xsk/rx.c @@ -25,6 +25,7 @@ static struct sk_buff *mlx5e_xsk_construct_skb(struct mlx5e_rq *rq, void *data, struct sk_buff *mlx5e_xsk_skb_from_cqe_mpwrq_linear(struct mlx5e_rq *rq, struct mlx5e_mpw_info *wi, + struct mlx5_cqe64 *cqe, u16 cqe_bcnt, u32 head_offset, u32 page_idx) diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/xsk/rx.h b/drivers/net/ethernet/mellanox/mlx5/core/en/xsk/rx.h index 7f88ccf67fdd..112c5b3ec165 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en/xsk/rx.h +++ b/drivers/net/ethernet/mellanox/mlx5/core/en/xsk/rx.h @@ -11,6 +11,7 @@ struct sk_buff *mlx5e_xsk_skb_from_cqe_mpwrq_linear(struct mlx5e_rq *rq, struct mlx5e_mpw_info *wi, + struct mlx5_cqe64 *cqe, u16 cqe_bcnt, u32 head_offset, u32 page_idx); diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp_rxtx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp_rxtx.c new file mode 100644 index 000000000000..31586f574fc0 --- /dev/null +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp_rxtx.c @@ -0,0 +1,248 @@ +// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB +/* Copyright (c) 2021 Mellanox Technologies. */ + +#include "en_accel/nvmeotcp_rxtx.h" +#include "en_accel/nvmeotcp.h" +#include + +#define MLX5E_TC_FLOW_ID_MASK 0x00ffffff +static void nvmeotcp_update_resync(struct mlx5e_nvmeotcp_queue *queue, + struct mlx5e_cqe128 *cqe128) +{ + const struct ulp_ddp_ulp_ops *ulp_ops; + u32 seq; + + seq = be32_to_cpu(cqe128->resync_tcp_sn); + ulp_ops = inet_csk(queue->sk)->icsk_ulp_ddp_ops; + if (ulp_ops && ulp_ops->resync_request) + ulp_ops->resync_request(queue->sk, seq, ULP_DDP_RESYNC_REQ); +} + +static void mlx5e_nvmeotcp_advance_sgl_iter(struct mlx5e_nvmeotcp_queue *queue) +{ + struct nvmeotcp_queue_entry *nqe = &queue->ccid_table[queue->ccid]; + + queue->ccoff += nqe->sgl[queue->ccsglidx].length; + queue->ccoff_inner = 0; + queue->ccsglidx++; +} + +static inline void +mlx5e_nvmeotcp_add_skb_frag(struct net_device *netdev, struct sk_buff *skb, + struct mlx5e_nvmeotcp_queue *queue, + struct nvmeotcp_queue_entry *nqe, u32 fragsz) +{ + dma_sync_single_for_cpu(&netdev->dev, + nqe->sgl[queue->ccsglidx].offset + queue->ccoff_inner, + fragsz, DMA_FROM_DEVICE); + page_ref_inc(compound_head(sg_page(&nqe->sgl[queue->ccsglidx]))); + // XXX: consider reducing the truesize, as no new memory is consumed + skb_add_rx_frag(skb, skb_shinfo(skb)->nr_frags, + sg_page(&nqe->sgl[queue->ccsglidx]), + nqe->sgl[queue->ccsglidx].offset + queue->ccoff_inner, + fragsz, + fragsz); +} + +static struct sk_buff* +mlx5_nvmeotcp_add_tail_nonlinear(struct mlx5e_nvmeotcp_queue *queue, + struct sk_buff *skb, skb_frag_t *org_frags, + int org_nr_frags, int frag_index) +{ + struct mlx5e_priv *priv = queue->priv; + + while (org_nr_frags != frag_index) { + if (skb_shinfo(skb)->nr_frags >= MAX_SKB_FRAGS) { + dev_kfree_skb_any(skb); + return NULL; + } + skb_add_rx_frag(skb, skb_shinfo(skb)->nr_frags, + skb_frag_page(&org_frags[frag_index]), + skb_frag_off(&org_frags[frag_index]), + skb_frag_size(&org_frags[frag_index]), + skb_frag_size(&org_frags[frag_index])); + page_ref_inc(skb_frag_page(&org_frags[frag_index])); + frag_index++; + } + return skb; +} + +static struct sk_buff* +mlx5_nvmeotcp_add_tail(struct mlx5e_nvmeotcp_queue *queue, struct sk_buff *skb, + int offset, int len) +{ + struct mlx5e_priv *priv = queue->priv; + + if (skb_shinfo(skb)->nr_frags >= MAX_SKB_FRAGS) { + dev_kfree_skb_any(skb); + return NULL; + } + skb_add_rx_frag(skb, skb_shinfo(skb)->nr_frags, + virt_to_page(skb->data), + offset, + len, + len); + page_ref_inc(virt_to_page(skb->data)); + return skb; +} + +static void mlx5_nvmeotcp_trim_nonlinear(struct sk_buff *skb, + skb_frag_t *org_frags, + int *frag_index, + int remaining) +{ + unsigned int frag_size; + int nr_frags; + + /* skip @remaining bytes in frags */ + *frag_index = 0; + while (remaining) { + frag_size = skb_frag_size(&skb_shinfo(skb)->frags[*frag_index]); + if (frag_size > remaining) { + skb_frag_off_add(&skb_shinfo(skb)->frags[*frag_index], + remaining); + skb_frag_size_sub(&skb_shinfo(skb)->frags[*frag_index], + remaining); + remaining = 0; + } else { + remaining -= frag_size; + skb_frag_unref(skb, *frag_index); + *frag_index += 1; + } + } + + /* save original frags for the tail and unref */ + nr_frags = skb_shinfo(skb)->nr_frags; + memcpy(&org_frags[*frag_index], &skb_shinfo(skb)->frags[*frag_index], + (nr_frags - *frag_index) * sizeof(skb_frag_t)); + while (--nr_frags >= *frag_index) + skb_frag_unref(skb, nr_frags); + + /* remove frags from skb */ + skb_shinfo(skb)->nr_frags = 0; + skb->len -= skb->data_len; + skb->truesize -= skb->data_len; + skb->data_len = 0; +} + +struct sk_buff* +mlx5e_nvmeotcp_handle_rx_skb(struct net_device *netdev, struct sk_buff *skb, + struct mlx5_cqe64 *cqe, u32 cqe_bcnt, + bool linear) +{ + int ccoff, cclen, hlen, ccid, remaining, fragsz, to_copy = 0; + struct mlx5e_priv *priv = netdev_priv(netdev); + skb_frag_t org_frags[MAX_SKB_FRAGS]; + struct mlx5e_nvmeotcp_queue *queue; + struct nvmeotcp_queue_entry *nqe; + int org_nr_frags, frag_index; + struct mlx5e_cqe128 *cqe128; + u32 queue_id; + + queue_id = (be32_to_cpu(cqe->sop_drop_qpn) & MLX5E_TC_FLOW_ID_MASK); + queue = mlx5e_nvmeotcp_get_queue(priv->nvmeotcp, queue_id); + if (unlikely(!queue)) { + dev_kfree_skb_any(skb); + return NULL; + } + + cqe128 = container_of(cqe, struct mlx5e_cqe128, cqe64); + if (cqe_is_nvmeotcp_resync(cqe)) { + nvmeotcp_update_resync(queue, cqe128); + mlx5e_nvmeotcp_put_queue(queue); + return skb; + } + + /* If a resync occurred in the previous cqe, + * the current cqe.crcvalid bit may not be valid, + * so we will treat it as 0 + */ + if (unlikely(queue->after_resync_cqe)) { + skb->ddp_crc = 0; + queue->after_resync_cqe = 0; + } else { + if (queue->crc_rx) + skb->ddp_crc = cqe_is_nvmeotcp_crcvalid(cqe); + else + skb->ddp_crc = cqe_is_nvmeotcp_zc(cqe); + } + + if (!cqe_is_nvmeotcp_zc(cqe)) { + mlx5e_nvmeotcp_put_queue(queue); + return skb; + } + + /* cc ddp from cqe */ + ccid = be16_to_cpu(cqe128->ccid); + ccoff = be32_to_cpu(cqe128->ccoff); + cclen = be16_to_cpu(cqe128->cclen); + hlen = be16_to_cpu(cqe128->hlen); + + /* carve a hole in the skb for DDP data */ + if (linear) { + skb_trim(skb, hlen); + } else { + org_nr_frags = skb_shinfo(skb)->nr_frags; + mlx5_nvmeotcp_trim_nonlinear(skb, org_frags, &frag_index, + cclen); + } + + nqe = &queue->ccid_table[ccid]; + + /* packet starts new ccid? */ + if (queue->ccid != ccid || queue->ccid_gen != nqe->ccid_gen) { + queue->ccid = ccid; + queue->ccoff = 0; + queue->ccoff_inner = 0; + queue->ccsglidx = 0; + queue->ccid_gen = nqe->ccid_gen; + } + + /* skip inside cc until the ccoff in the cqe */ + while (queue->ccoff + queue->ccoff_inner < ccoff) { + remaining = nqe->sgl[queue->ccsglidx].length - queue->ccoff_inner; + fragsz = min_t(off_t, remaining, + ccoff - (queue->ccoff + queue->ccoff_inner)); + + if (fragsz == remaining) + mlx5e_nvmeotcp_advance_sgl_iter(queue); + else + queue->ccoff_inner += fragsz; + } + + /* adjust the skb according to the cqe cc */ + while (to_copy < cclen) { + if (skb_shinfo(skb)->nr_frags >= MAX_SKB_FRAGS) { + dev_kfree_skb_any(skb); + mlx5e_nvmeotcp_put_queue(queue); + return NULL; + } + + remaining = nqe->sgl[queue->ccsglidx].length - queue->ccoff_inner; + fragsz = min_t(int, remaining, cclen - to_copy); + + mlx5e_nvmeotcp_add_skb_frag(netdev, skb, queue, nqe, fragsz); + to_copy += fragsz; + if (fragsz == remaining) + mlx5e_nvmeotcp_advance_sgl_iter(queue); + else + queue->ccoff_inner += fragsz; + } + + if (cqe_bcnt > hlen + cclen) { + remaining = cqe_bcnt - hlen - cclen; + if (linear) + skb = mlx5_nvmeotcp_add_tail(queue, skb, + offset_in_page(skb->data) + + hlen + cclen, + remaining); + else + skb = mlx5_nvmeotcp_add_tail_nonlinear(queue, skb, + org_frags, + org_nr_frags, + frag_index); + } + + mlx5e_nvmeotcp_put_queue(queue); + return skb; +} diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp_rxtx.h b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp_rxtx.h new file mode 100644 index 000000000000..65456b46c33f --- /dev/null +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp_rxtx.h @@ -0,0 +1,43 @@ +/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */ +/* Copyright (c) 2021 Mellanox Technologies. */ +#ifndef __MLX5E_NVMEOTCP_RXTX_H__ +#define __MLX5E_NVMEOTCP_RXTX_H__ + +#ifdef CONFIG_MLX5_EN_NVMEOTCP + +#include +#include "en.h" + +struct sk_buff* +mlx5e_nvmeotcp_handle_rx_skb(struct net_device *netdev, struct sk_buff *skb, + struct mlx5_cqe64 *cqe, u32 cqe_bcnt, bool linear); + +static inline int mlx5_nvmeotcp_get_headlen(struct mlx5_cqe64 *cqe, u32 cqe_bcnt) +{ + struct mlx5e_cqe128 *cqe128; + + if (!cqe_is_nvmeotcp_zc(cqe) || cqe_is_nvmeotcp_resync(cqe)) + return cqe_bcnt; + + cqe128 = container_of(cqe, struct mlx5e_cqe128, cqe64); + return be16_to_cpu(cqe128->hlen); +} + +#else +static inline struct sk_buff* +mlx5e_nvmeotcp_handle_rx_skb(struct net_device *netdev, struct sk_buff *skb, + struct mlx5_cqe64 *cqe, u32 cqe_bcnt, bool linear) +{ return skb; } + +static inline int mlx5_nvmeotcp_get_headlen(struct mlx5_cqe64 *cqe, u32 cqe_bcnt) +{ return cqe_bcnt; } + +#endif /* CONFIG_MLX5_EN_NVMEOTCP */ + +static inline u16 mlx5e_get_headlen_hint(struct mlx5_cqe64 *cqe, u32 cqe_bcnt) +{ + return min_t(u32, MLX5E_RX_MAX_HEAD, mlx5_nvmeotcp_get_headlen(cqe, cqe_bcnt)); +} + + +#endif /* __MLX5E_NVMEOTCP_RXTX_H__ */ diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c index e92dd4666955..8a20a5800e20 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c @@ -48,6 +48,7 @@ #include "en_accel/ipsec_rxtx.h" #include "en_accel/tls_rxtx.h" #include "en_accel/nvmeotcp.h" +#include "en_accel/nvmeotcp_rxtx.h" #include "en/xdp.h" #include "en/xsk/rx.h" #include "en/health.h" @@ -57,9 +58,11 @@ static struct sk_buff * mlx5e_skb_from_cqe_mpwrq_linear(struct mlx5e_rq *rq, struct mlx5e_mpw_info *wi, + struct mlx5_cqe64 *cqe, u16 cqe_bcnt, u32 head_offset, u32 page_idx); static struct sk_buff * mlx5e_skb_from_cqe_mpwrq_nonlinear(struct mlx5e_rq *rq, struct mlx5e_mpw_info *wi, + struct mlx5_cqe64 *cqe, u16 cqe_bcnt, u32 head_offset, u32 page_idx); static void mlx5e_handle_rx_cqe(struct mlx5e_rq *rq, struct mlx5_cqe64 *cqe); static void mlx5e_handle_rx_cqe_mpwrq(struct mlx5e_rq *rq, struct mlx5_cqe64 *cqe); @@ -1186,6 +1189,12 @@ mlx5e_skb_from_cqe_linear(struct mlx5e_rq *rq, struct mlx5_cqe64 *cqe, /* queue up for recycling/reuse */ page_ref_inc(di->page); +#ifdef CONFIG_MLX5_EN_NVMEOTCP + if (cqe_is_nvmeotcp(cqe)) + skb = mlx5e_nvmeotcp_handle_rx_skb(rq->netdev, skb, cqe, + cqe_bcnt, true); +#endif + return skb; } @@ -1194,8 +1203,8 @@ mlx5e_skb_from_cqe_nonlinear(struct mlx5e_rq *rq, struct mlx5_cqe64 *cqe, struct mlx5e_wqe_frag_info *wi, u32 cqe_bcnt) { struct mlx5e_rq_frag_info *frag_info = &rq->wqe.info.arr[0]; + u16 headlen = mlx5e_get_headlen_hint(cqe, cqe_bcnt); struct mlx5e_wqe_frag_info *head_wi = wi; - u16 headlen = min_t(u32, MLX5E_RX_MAX_HEAD, cqe_bcnt); u16 frag_headlen = headlen; u16 byte_cnt = cqe_bcnt - headlen; struct sk_buff *skb; @@ -1204,7 +1213,7 @@ mlx5e_skb_from_cqe_nonlinear(struct mlx5e_rq *rq, struct mlx5_cqe64 *cqe, * might spread among multiple pages. */ skb = napi_alloc_skb(rq->cq.napi, - ALIGN(MLX5E_RX_MAX_HEAD, sizeof(long))); + ALIGN(headlen, sizeof(long))); if (unlikely(!skb)) { rq->stats->buff_alloc_err++; return NULL; @@ -1230,6 +1239,12 @@ mlx5e_skb_from_cqe_nonlinear(struct mlx5e_rq *rq, struct mlx5_cqe64 *cqe, skb->tail += headlen; skb->len += headlen; +#ifdef CONFIG_MLX5_EN_NVMEOTCP + if (cqe_is_nvmeotcp(cqe)) + skb = mlx5e_nvmeotcp_handle_rx_skb(rq->netdev, skb, cqe, + cqe_bcnt, false); +#endif + return skb; } @@ -1389,7 +1404,7 @@ static void mlx5e_handle_rx_cqe_mpwrq_rep(struct mlx5e_rq *rq, struct mlx5_cqe64 skb = INDIRECT_CALL_2(rq->mpwqe.skb_from_cqe_mpwrq, mlx5e_skb_from_cqe_mpwrq_linear, mlx5e_skb_from_cqe_mpwrq_nonlinear, - rq, wi, cqe_bcnt, head_offset, page_idx); + rq, wi, cqe, cqe_bcnt, head_offset, page_idx); if (!skb) goto mpwrq_cqe_out; @@ -1423,17 +1438,18 @@ const struct mlx5e_rx_handlers mlx5e_rx_handlers_rep = { static struct sk_buff * mlx5e_skb_from_cqe_mpwrq_nonlinear(struct mlx5e_rq *rq, struct mlx5e_mpw_info *wi, + struct mlx5_cqe64 *cqe, u16 cqe_bcnt, u32 head_offset, u32 page_idx) { - u16 headlen = min_t(u16, MLX5E_RX_MAX_HEAD, cqe_bcnt); struct mlx5e_dma_info *di = &wi->umr.dma_info[page_idx]; + u16 headlen = mlx5e_get_headlen_hint(cqe, cqe_bcnt); u32 frag_offset = head_offset + headlen; u32 byte_cnt = cqe_bcnt - headlen; struct mlx5e_dma_info *head_di = di; struct sk_buff *skb; skb = napi_alloc_skb(rq->cq.napi, - ALIGN(MLX5E_RX_MAX_HEAD, sizeof(long))); + ALIGN(headlen, sizeof(long))); if (unlikely(!skb)) { rq->stats->buff_alloc_err++; return NULL; @@ -1464,11 +1480,18 @@ mlx5e_skb_from_cqe_mpwrq_nonlinear(struct mlx5e_rq *rq, struct mlx5e_mpw_info *w skb->tail += headlen; skb->len += headlen; +#ifdef CONFIG_MLX5_EN_NVMEOTCP + if (cqe_is_nvmeotcp(cqe)) + skb = mlx5e_nvmeotcp_handle_rx_skb(rq->netdev, skb, cqe, + cqe_bcnt, false); +#endif + return skb; } static struct sk_buff * mlx5e_skb_from_cqe_mpwrq_linear(struct mlx5e_rq *rq, struct mlx5e_mpw_info *wi, + struct mlx5_cqe64 *cqe, u16 cqe_bcnt, u32 head_offset, u32 page_idx) { struct mlx5e_dma_info *di = &wi->umr.dma_info[page_idx]; @@ -1510,6 +1533,12 @@ mlx5e_skb_from_cqe_mpwrq_linear(struct mlx5e_rq *rq, struct mlx5e_mpw_info *wi, /* queue up for recycling/reuse */ page_ref_inc(di->page); +#ifdef CONFIG_MLX5_EN_NVMEOTCP + if (cqe_is_nvmeotcp(cqe)) + skb = mlx5e_nvmeotcp_handle_rx_skb(rq->netdev, skb, cqe, + cqe_bcnt, true); +#endif + return skb; } @@ -1548,7 +1577,7 @@ static void mlx5e_handle_rx_cqe_mpwrq(struct mlx5e_rq *rq, struct mlx5_cqe64 *cq skb = INDIRECT_CALL_2(rq->mpwqe.skb_from_cqe_mpwrq, mlx5e_skb_from_cqe_mpwrq_linear, mlx5e_skb_from_cqe_mpwrq_nonlinear, - rq, wi, cqe_bcnt, head_offset, page_idx); + rq, wi, cqe, cqe_bcnt, head_offset, page_idx); if (!skb) goto mpwrq_cqe_out; From patchwork Thu Jul 22 11:03:08 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Boris Pismenny X-Patchwork-Id: 485103 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-20.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER, INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4784CC6377D for ; Thu, 22 Jul 2021 11:05:55 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 30F1D6120D for ; Thu, 22 Jul 2021 11:05:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231721AbhGVKZT (ORCPT ); Thu, 22 Jul 2021 06:25:19 -0400 Received: from mail-bn8nam12on2089.outbound.protection.outlook.com ([40.107.237.89]:58592 "EHLO NAM12-BN8-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S231698AbhGVKZM (ORCPT ); Thu, 22 Jul 2021 06:25:12 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=PJ7dlShJPgkqnGjlAH4WSYaw+oCZda6H9xqqQxg+gRxhbSHJvGxGnvsqvHiuMKlZy5CGYzuJS7UMXPl3dspjX32V8PA+GIiDY2sbZaWz8hMlIQ6nv72+Dsu0zfCJAJ8dicIADlrcfO5l1i2yIOsTHseK813D4OUtEFKtmjak9odDXQ4XULSz5zolTxKh6Ka6ToVC/VpeXkRVwDmi3WVFXtgnE0Ykdb4wwk0xAH1J5GjDbYs/ZgQrrpjVtzKvaaFDI56hOyO6zjzcdZRyBesIuz6YZv9tf4af2s02o2GNgv00X2jF5OJgAiIOYfuUtdWxb6KolsFSbbCnU7n3ZVg+Jw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=JfQ8fSajKUipAvFvd3MML2Gi+7jl0rWgHEIBjaybNtE=; b=XNQhwlHTjwiWJ8uC6dNfEoHoFm3y/PlJ1ItARtQOdtwaNlJvs59XIT/Kg+IJern0uwvb3MtWQeawsca74LW2OqUKz+zHGY0jWAjwnc+7e7mOF4xnj6APzdWE1qk9gd0sP9anju/7kAZ6JWIKpVM6CUV3ZMOb8Euqc0jwvcF1+SqDNsvJ+oKmfEwc487lRQ8errL17uMsak4Dofx8MRL2VnX63cuBBUDZSVIoBsTm4KcHy5wCmuiwPCx5oF28KXfc1ZCiNeVLLBSlSgn375f6ga8CnYhRb0y/yM23mCzJtRduau9ZU+BarJV0Z20+iv3oVZJGeyuLk5/DjHEz+Cg2Xw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.32) smtp.rcpttodomain=marvell.com smtp.mailfrom=nvidia.com; dmarc=pass (p=quarantine sp=none pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=JfQ8fSajKUipAvFvd3MML2Gi+7jl0rWgHEIBjaybNtE=; b=aVLET+UBdTNwLlhVv4ELlfdXe2d2Yh7XJSbQZXx1J2/W9JwWNVjR64IZILu1ZraSIR/Vu8m2MBCf9S72WNtlnoz6q/98vXpz0oPaFifSFFElepg9+RnMFWRf0xFSoFCHyOBasNJJ77kLAYkgiS7Q6MKM3T8ZrdHGSqUW55+y6KG202+nSQlAE6syqcv3NalFYJt6raNTkjSEwKEG7Oxg37ibZ/pnJJV1OfQGdb4YECUJPx11oxGnoi023a+LWAxI+sciW2FepabZHBFkuVhaWv3IfTXcs01qjooh5C1v7dwZCfM4JWOBBeYbNVV39eFywA45Zano7mVCOGjf0TUH6g== Received: from DM6PR02CA0090.namprd02.prod.outlook.com (2603:10b6:5:1f4::31) by BYAPR12MB2949.namprd12.prod.outlook.com (2603:10b6:a03:12f::24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4331.31; Thu, 22 Jul 2021 11:05:46 +0000 Received: from DM6NAM11FT022.eop-nam11.prod.protection.outlook.com (2603:10b6:5:1f4:cafe::57) by DM6PR02CA0090.outlook.office365.com (2603:10b6:5:1f4::31) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4352.25 via Frontend Transport; Thu, 22 Jul 2021 11:05:46 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.32) smtp.mailfrom=nvidia.com; marvell.com; dkim=none (message not signed) header.d=none; marvell.com; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.32 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.32; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.32) by DM6NAM11FT022.mail.protection.outlook.com (10.13.172.210) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4352.24 via Frontend Transport; Thu, 22 Jul 2021 11:05:46 +0000 Received: from HQMAIL107.nvidia.com (172.20.187.13) by HQMAIL109.nvidia.com (172.20.187.15) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Thu, 22 Jul 2021 04:05:45 -0700 Received: from vdi.nvidia.com (172.20.187.5) by mail.nvidia.com (172.20.187.13) with Microsoft SMTP Server id 15.0.1497.2 via Frontend Transport; Thu, 22 Jul 2021 11:05:41 +0000 From: Boris Pismenny To: , , , , , , , , , , CC: , , , , , , Boris Pismenny , Or Gerlitz , Yoray Zack Subject: [PATCH v5 net-next 19/36] net/mlx5e: NVMEoTCP statistics Date: Thu, 22 Jul 2021 14:03:08 +0300 Message-ID: <20210722110325.371-20-borisp@nvidia.com> X-Mailer: git-send-email 2.24.1 In-Reply-To: <20210722110325.371-1-borisp@nvidia.com> References: <20210722110325.371-1-borisp@nvidia.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 73611479-f08c-4c9f-59ca-08d94d00a7e3 X-MS-TrafficTypeDiagnostic: BYAPR12MB2949: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:126; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: HId/qNG9yiSZ8gFpGPCe7BexEBxldsOLqhWgl3+gt4c0EF871moRlQbTyV/He9Fa/Z8ShA9FhxkOxdZXekiPck4BfoRxFLLrPtjBHEP4+mhQMFIY0Ilybi6o/jNxlDE7N41WrV+pQw+T26ReI+Mzt+xtPSrUC4iyhOo8xI1q1eKQVaRTtuPQkEbozUR1C4uVyUG8DHI9TZgYO6KYgqux6u+DUq+mGGhPjuqf925ivM2zVUcnc2ucigq/sQTRFi/X0HK6gX76SKRG+DIidbdAiWFtORKBPjL5uK08IFUrUhs24YS23U23hmHbE3ueNrmY5nCFp0fctV1wLl6pi7pQMRbApmL2ZXXg6zlDtMUOi3oRWf1WC/KcEkUHVlIETUA0ryBU/EHViz1OldoMrzjOpZmmXDeOglBL16hckhQL3UOvAHmZYBsKsY4ysGb//CQDdKlkFTjvoaJUj74dnObAe3mDj7SsM+YarrYD7ElvvS/Y7fZQ4fRPQ9Si1l5t8Su93oHTsyutbZ742O2WaBUl1cI56GRx14UA3wVMnGgttAOwAZbSRHT2xhN2A75JiHTDl/VWXcQ+WgMu9o52YgFRHYMRsHR7MNJ0DlqFAOiAZknLj+SJ+yfNWB66TEPTkXsEQJ/+H44aOOyj2oNTdNJhO8dNgmS03HdR79DaxUFJITdeKfYCmEHmxBwEaSlX9oenlLT8ngTVxgq0OWRXIxEAjVCa15YpLPYPmpmCb7WL5oE= X-Forefront-Antispam-Report: CIP:216.228.112.32; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:schybrid01.nvidia.com; CAT:NONE; SFS:(4636009)(136003)(376002)(39860400002)(346002)(396003)(46966006)(36840700001)(921005)(7636003)(70586007)(82310400003)(4326008)(30864003)(86362001)(107886003)(8676002)(1076003)(2906002)(82740400003)(336012)(186003)(5660300002)(6666004)(356005)(70206006)(7696005)(7416002)(83380400001)(8936002)(478600001)(426003)(26005)(47076005)(2616005)(36756003)(54906003)(36860700001)(110136005)(316002); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Jul 2021 11:05:46.0110 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 73611479-f08c-4c9f-59ca-08d94d00a7e3 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.112.32]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DM6NAM11FT022.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR12MB2949 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Ben Ben-Ishay NVMEoTCP offload statistics includes both control and data path statistic: counters for ndo, offloaded packets/bytes, dropped packets and resync operation. Signed-off-by: Boris Pismenny Signed-off-by: Ben Ben-Ishay Signed-off-by: Or Gerlitz Signed-off-by: Yoray Zack --- .../mellanox/mlx5/core/en_accel/nvmeotcp.c | 23 +++++++++++- .../mlx5/core/en_accel/nvmeotcp_rxtx.c | 16 ++++++++ .../ethernet/mellanox/mlx5/core/en_stats.c | 37 +++++++++++++++++++ .../ethernet/mellanox/mlx5/core/en_stats.h | 24 ++++++++++++ 4 files changed, 98 insertions(+), 2 deletions(-) diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c index 4fdfbe4468ee..7f6607cac9fb 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c @@ -667,6 +667,11 @@ mlx5e_nvmeotcp_queue_init(struct net_device *netdev, struct mlx5_core_dev *mdev = priv->mdev; struct mlx5e_nvmeotcp_queue *queue; int max_wqe_sz_cap, queue_id, err; + struct mlx5e_rq_stats *stats; + u32 channel_ix; + + channel_ix = mlx5e_get_channel_ix_from_io_cpu(priv, config->io_cpu); + stats = &priv->channel_stats[channel_ix].rq; if (tconfig->type != ULP_DDP_NVME) { err = -EOPNOTSUPP; @@ -694,8 +699,7 @@ mlx5e_nvmeotcp_queue_init(struct net_device *netdev, queue->id = queue_id; queue->dgst = config->dgst; queue->pda = config->cpda; - queue->channel_ix = mlx5e_get_channel_ix_from_io_cpu(priv, - config->io_cpu); + queue->channel_ix = channel_ix; queue->size = config->queue_size; max_wqe_sz_cap = min_t(int, MAX_DS_VALUE * MLX5_SEND_WQE_DS, MLX5_CAP_GEN(mdev, max_wqe_sz_sq) << OCTWORD_SHIFT); @@ -715,6 +719,7 @@ mlx5e_nvmeotcp_queue_init(struct net_device *netdev, if (err) goto destroy_rx; + stats->nvmeotcp_queue_init++; write_lock_bh(&sk->sk_callback_lock); ulp_ddp_set_ctx(sk, queue); write_unlock_bh(&sk->sk_callback_lock); @@ -729,6 +734,7 @@ mlx5e_nvmeotcp_queue_init(struct net_device *netdev, free_queue: kfree(queue); out: + stats->nvmeotcp_queue_init_fail++; return err; } @@ -739,11 +745,15 @@ mlx5e_nvmeotcp_queue_teardown(struct net_device *netdev, struct mlx5e_priv *priv = netdev_priv(netdev); struct mlx5_core_dev *mdev = priv->mdev; struct mlx5e_nvmeotcp_queue *queue; + struct mlx5e_rq_stats *stats; queue = container_of(ulp_ddp_get_ctx(sk), struct mlx5e_nvmeotcp_queue, ulp_ddp_ctx); napi_synchronize(&priv->channels.c[queue->channel_ix]->napi); + stats = &priv->channel_stats[queue->channel_ix].rq; + stats->nvmeotcp_queue_teardown++; + WARN_ON(refcount_read(&queue->ref_count) != 1); if (queue->zerocopy | queue->crc_rx) mlx5e_nvmeotcp_destroy_rx(queue, mdev, queue->zerocopy); @@ -765,6 +775,7 @@ mlx5e_nvmeotcp_ddp_setup(struct net_device *netdev, struct mlx5e_priv *priv = netdev_priv(netdev); struct scatterlist *sg = ddp->sg_table.sgl; struct mlx5e_nvmeotcp_queue *queue; + struct mlx5e_rq_stats *stats; struct mlx5_core_dev *mdev; int i, size = 0, count = 0; @@ -786,6 +797,11 @@ mlx5e_nvmeotcp_ddp_setup(struct net_device *netdev, queue->ccid_table[ddp->command_id].ccid_gen++; queue->ccid_table[ddp->command_id].sgl_length = count; + stats = &priv->channel_stats[queue->channel_ix].rq; + stats->nvmeotcp_ddp_setup++; + if (unlikely(mlx5e_nvmeotcp_post_klm_wqe(queue, KLM_UMR, ddp->command_id, count))) + stats->nvmeotcp_ddp_setup_fail++; + return 0; } @@ -826,6 +842,7 @@ mlx5e_nvmeotcp_ddp_teardown(struct net_device *netdev, struct mlx5e_nvmeotcp_queue *queue; struct mlx5e_priv *priv = netdev_priv(netdev); struct nvmeotcp_queue_entry *q_entry; + struct mlx5e_rq_stats *stats; queue = container_of(ulp_ddp_get_ctx(sk), struct mlx5e_nvmeotcp_queue, ulp_ddp_ctx); q_entry = &queue->ccid_table[ddp->command_id]; @@ -835,6 +852,8 @@ mlx5e_nvmeotcp_ddp_teardown(struct net_device *netdev, q_entry->queue = queue; mlx5e_nvmeotcp_post_klm_wqe(queue, KLM_UMR, ddp->command_id, 0); + stats = &priv->channel_stats[queue->channel_ix].rq; + stats->nvmeotcp_ddp_teardown++; return 0; } diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp_rxtx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp_rxtx.c index 31586f574fc0..d4ac914e2a5b 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp_rxtx.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp_rxtx.c @@ -10,12 +10,16 @@ static void nvmeotcp_update_resync(struct mlx5e_nvmeotcp_queue *queue, struct mlx5e_cqe128 *cqe128) { const struct ulp_ddp_ulp_ops *ulp_ops; + struct mlx5e_rq_stats *stats; u32 seq; seq = be32_to_cpu(cqe128->resync_tcp_sn); ulp_ops = inet_csk(queue->sk)->icsk_ulp_ddp_ops; if (ulp_ops && ulp_ops->resync_request) ulp_ops->resync_request(queue->sk, seq, ULP_DDP_RESYNC_REQ); + + stats = queue->priv->channels.c[queue->channel_ix]->rq.stats; + stats->nvmeotcp_resync++; } static void mlx5e_nvmeotcp_advance_sgl_iter(struct mlx5e_nvmeotcp_queue *queue) @@ -50,10 +54,13 @@ mlx5_nvmeotcp_add_tail_nonlinear(struct mlx5e_nvmeotcp_queue *queue, int org_nr_frags, int frag_index) { struct mlx5e_priv *priv = queue->priv; + struct mlx5e_rq_stats *stats; while (org_nr_frags != frag_index) { if (skb_shinfo(skb)->nr_frags >= MAX_SKB_FRAGS) { dev_kfree_skb_any(skb); + stats = priv->channels.c[queue->channel_ix]->rq.stats; + stats->nvmeotcp_drop++; return NULL; } skb_add_rx_frag(skb, skb_shinfo(skb)->nr_frags, @@ -72,9 +79,12 @@ mlx5_nvmeotcp_add_tail(struct mlx5e_nvmeotcp_queue *queue, struct sk_buff *skb, int offset, int len) { struct mlx5e_priv *priv = queue->priv; + struct mlx5e_rq_stats *stats; if (skb_shinfo(skb)->nr_frags >= MAX_SKB_FRAGS) { dev_kfree_skb_any(skb); + stats = priv->channels.c[queue->channel_ix]->rq.stats; + stats->nvmeotcp_drop++; return NULL; } skb_add_rx_frag(skb, skb_shinfo(skb)->nr_frags, @@ -135,6 +145,7 @@ mlx5e_nvmeotcp_handle_rx_skb(struct net_device *netdev, struct sk_buff *skb, skb_frag_t org_frags[MAX_SKB_FRAGS]; struct mlx5e_nvmeotcp_queue *queue; struct nvmeotcp_queue_entry *nqe; + struct mlx5e_rq_stats *stats; int org_nr_frags, frag_index; struct mlx5e_cqe128 *cqe128; u32 queue_id; @@ -172,6 +183,8 @@ mlx5e_nvmeotcp_handle_rx_skb(struct net_device *netdev, struct sk_buff *skb, return skb; } + stats = priv->channels.c[queue->channel_ix]->rq.stats; + /* cc ddp from cqe */ ccid = be16_to_cpu(cqe128->ccid); ccoff = be32_to_cpu(cqe128->ccoff); @@ -214,6 +227,7 @@ mlx5e_nvmeotcp_handle_rx_skb(struct net_device *netdev, struct sk_buff *skb, while (to_copy < cclen) { if (skb_shinfo(skb)->nr_frags >= MAX_SKB_FRAGS) { dev_kfree_skb_any(skb); + stats->nvmeotcp_drop++; mlx5e_nvmeotcp_put_queue(queue); return NULL; } @@ -243,6 +257,8 @@ mlx5e_nvmeotcp_handle_rx_skb(struct net_device *netdev, struct sk_buff *skb, frag_index); } + stats->nvmeotcp_offload_packets++; + stats->nvmeotcp_offload_bytes += cclen; mlx5e_nvmeotcp_put_queue(queue); return skb; } diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_stats.c b/drivers/net/ethernet/mellanox/mlx5/core/en_stats.c index e4f5b6395148..3d95e46422e5 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_stats.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_stats.c @@ -34,6 +34,7 @@ #include "en.h" #include "en_accel/tls.h" #include "en_accel/en_accel.h" +#include "en_accel/nvmeotcp.h" static unsigned int stats_grps_num(struct mlx5e_priv *priv) { @@ -187,6 +188,18 @@ static const struct counter_desc sw_stats_desc[] = { { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_tls_resync_res_retry) }, { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_tls_resync_res_skip) }, { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_tls_err) }, +#endif +#ifdef CONFIG_MLX5_EN_NVMEOTCP + { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_nvmeotcp_queue_init) }, + { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_nvmeotcp_queue_init_fail) }, + { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_nvmeotcp_queue_teardown) }, + { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_nvmeotcp_ddp_setup) }, + { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_nvmeotcp_ddp_setup_fail) }, + { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_nvmeotcp_ddp_teardown) }, + { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_nvmeotcp_drop) }, + { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_nvmeotcp_resync) }, + { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_nvmeotcp_offload_packets) }, + { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_nvmeotcp_offload_bytes) }, #endif { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, ch_events) }, { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, ch_poll) }, @@ -349,6 +362,18 @@ static void mlx5e_stats_grp_sw_update_stats_rq_stats(struct mlx5e_sw_stats *s, s->rx_tls_resync_res_skip += rq_stats->tls_resync_res_skip; s->rx_tls_err += rq_stats->tls_err; #endif +#ifdef CONFIG_MLX5_EN_NVMEOTCP + s->rx_nvmeotcp_queue_init += rq_stats->nvmeotcp_queue_init; + s->rx_nvmeotcp_queue_init_fail += rq_stats->nvmeotcp_queue_init_fail; + s->rx_nvmeotcp_queue_teardown += rq_stats->nvmeotcp_queue_teardown; + s->rx_nvmeotcp_ddp_setup += rq_stats->nvmeotcp_ddp_setup; + s->rx_nvmeotcp_ddp_setup_fail += rq_stats->nvmeotcp_ddp_setup_fail; + s->rx_nvmeotcp_ddp_teardown += rq_stats->nvmeotcp_ddp_teardown; + s->rx_nvmeotcp_drop += rq_stats->nvmeotcp_drop; + s->rx_nvmeotcp_resync += rq_stats->nvmeotcp_resync; + s->rx_nvmeotcp_offload_packets += rq_stats->nvmeotcp_offload_packets; + s->rx_nvmeotcp_offload_bytes += rq_stats->nvmeotcp_offload_bytes; +#endif } static void mlx5e_stats_grp_sw_update_stats_ch_stats(struct mlx5e_sw_stats *s, @@ -1788,6 +1813,18 @@ static const struct counter_desc rq_stats_desc[] = { { MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, tls_resync_res_skip) }, { MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, tls_err) }, #endif +#ifdef CONFIG_MLX5_EN_NVMEOTCP + { MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, nvmeotcp_queue_init) }, + { MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, nvmeotcp_queue_init_fail) }, + { MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, nvmeotcp_queue_teardown) }, + { MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, nvmeotcp_ddp_setup) }, + { MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, nvmeotcp_ddp_setup_fail) }, + { MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, nvmeotcp_ddp_teardown) }, + { MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, nvmeotcp_drop) }, + { MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, nvmeotcp_resync) }, + { MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, nvmeotcp_offload_packets) }, + { MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, nvmeotcp_offload_bytes) }, +#endif }; static const struct counter_desc sq_stats_desc[] = { diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_stats.h b/drivers/net/ethernet/mellanox/mlx5/core/en_stats.h index 139e59f30db0..e645ee83de97 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_stats.h +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_stats.h @@ -194,6 +194,18 @@ struct mlx5e_sw_stats { u64 rx_congst_umr; u64 rx_arfs_err; u64 rx_recover; +#ifdef CONFIG_MLX5_EN_NVMEOTCP + u64 rx_nvmeotcp_queue_init; + u64 rx_nvmeotcp_queue_init_fail; + u64 rx_nvmeotcp_queue_teardown; + u64 rx_nvmeotcp_ddp_setup; + u64 rx_nvmeotcp_ddp_setup_fail; + u64 rx_nvmeotcp_ddp_teardown; + u64 rx_nvmeotcp_drop; + u64 rx_nvmeotcp_resync; + u64 rx_nvmeotcp_offload_packets; + u64 rx_nvmeotcp_offload_bytes; +#endif u64 ch_events; u64 ch_poll; u64 ch_arm; @@ -354,6 +366,18 @@ struct mlx5e_rq_stats { u64 tls_resync_res_skip; u64 tls_err; #endif +#ifdef CONFIG_MLX5_EN_NVMEOTCP + u64 nvmeotcp_queue_init; + u64 nvmeotcp_queue_init_fail; + u64 nvmeotcp_queue_teardown; + u64 nvmeotcp_ddp_setup; + u64 nvmeotcp_ddp_setup_fail; + u64 nvmeotcp_ddp_teardown; + u64 nvmeotcp_drop; + u64 nvmeotcp_resync; + u64 nvmeotcp_offload_packets; + u64 nvmeotcp_offload_bytes; +#endif }; struct mlx5e_sq_stats { From patchwork Thu Jul 22 11:03:09 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Boris Pismenny X-Patchwork-Id: 484316 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-20.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER, INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 73794C63793 for ; Thu, 22 Jul 2021 11:05:57 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 5D8AB60725 for ; Thu, 22 Jul 2021 11:05:57 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231792AbhGVKZV (ORCPT ); Thu, 22 Jul 2021 06:25:21 -0400 Received: from mail-dm6nam10on2053.outbound.protection.outlook.com ([40.107.93.53]:13281 "EHLO NAM10-DM6-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S231635AbhGVKZS (ORCPT ); Thu, 22 Jul 2021 06:25:18 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=TcFEkcgFx1iuI6jCJv2stPejo+wNVaATGhANHDeYxeh6SpkqZVMaF4Q/58rifi+Vd8tcv2HF16hJDZ2pZdLIQF9D7pbaDCXt+aQiHIl7xfDt1qL3vFFZzAmejw86V4QNj3DoQvCQqoT21mOjp5MTmQInX4cZBI/qaLt4cJ7Sr+6In3C8uT0Eu85CBqnWZI1nsqGpq3Ui1EV3M3x36AjqxS7/UCNEYWbetU60Q5aRJnzAuMZNnmDzAoriMx5qo+IdVDo9HOHBmvfYvEI98K9cOjQrqFViBVytV2W96fKszHf0pEY9sIh2qQvv+gEVz+VVu9PU8MDY6iKx9MKM0MHfEg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=f9zgKlqMeNwmq8uZAGyRhF73sJg5UEMwSoT/sBqATis=; b=h8S//K6SSJtdEgDfVk+b3Tw7LaAU37yOn/YazepWKoTcGNiS1r2o0aW9ImpYJqTbIdVf5de90ndgR6ZRfx8HHGr2Ae7A+/+3FRkT0G74HE9/GQB0+IjxkztHSc/RveTpQzPv1UJhvAzIPIATMv3Yh8NDkTaoYNYGzG5seYwN6hb2MB10RTw/TC4zxkjbhj9udAfwk437m0sl3NXZo3V/pz/vryrUSnLEiuvniqiQpVudF7yu0izU2ggs28DXglNeQNp+rQsFBn4y1m3Zk0g4q/IcOmyS5yqWyYy3eHGpFsdju82rCfUb+/G+gBjEoGPHOH8s9FtuBzhG8FpdUFFomA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.34) smtp.rcpttodomain=google.com smtp.mailfrom=nvidia.com; dmarc=pass (p=quarantine sp=none pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=f9zgKlqMeNwmq8uZAGyRhF73sJg5UEMwSoT/sBqATis=; b=fMlheNTUERBZfiduXBq9CAbBkHEpdegts6qPzEsXPny4uUyGLKpewHRPQWZ2Gn4CL24o8VYkNeKAnB3FsI6wbs3izsnSDaY9H2aQQ3hfTuilya23WGwggspdMpaFvk+8BdEHEi5guNPpoBM0iUQPeJbIuXDqPSTQV8wvKEI/Nfylk8XAt9ui5dySH3jBQP7Ht3zM/rSdSeXQlleTr3x0ncmB5BHVpxss1s5gzIL/xQboriir8rG/D8VlXoN8GI4beHf+HythWbWU6LK6Cn52gjCoQP7+ZiAsonKO1fvAArYgDi3dGDJ7LTtTTX9+zEmpnxs/rxkX6PK5RGZLddLpig== Received: from BN6PR1201CA0014.namprd12.prod.outlook.com (2603:10b6:405:4c::24) by BN8PR12MB3251.namprd12.prod.outlook.com (2603:10b6:408:9b::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4352.25; Thu, 22 Jul 2021 11:05:51 +0000 Received: from BN8NAM11FT065.eop-nam11.prod.protection.outlook.com (2603:10b6:405:4c:cafe::3e) by BN6PR1201CA0014.outlook.office365.com (2603:10b6:405:4c::24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4352.24 via Frontend Transport; Thu, 22 Jul 2021 11:05:51 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.34) smtp.mailfrom=nvidia.com; google.com; dkim=none (message not signed) header.d=none; google.com; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.34 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.34; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.34) by BN8NAM11FT065.mail.protection.outlook.com (10.13.177.63) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4352.24 via Frontend Transport; Thu, 22 Jul 2021 11:05:51 +0000 Received: from HQMAIL105.nvidia.com (172.20.187.12) by HQMAIL107.nvidia.com (172.20.187.13) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Thu, 22 Jul 2021 11:05:50 +0000 Received: from HQMAIL107.nvidia.com (172.20.187.13) by HQMAIL105.nvidia.com (172.20.187.12) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Thu, 22 Jul 2021 11:05:50 +0000 Received: from vdi.nvidia.com (172.20.187.5) by mail.nvidia.com (172.20.187.13) with Microsoft SMTP Server id 15.0.1497.2 via Frontend Transport; Thu, 22 Jul 2021 11:05:45 +0000 From: Boris Pismenny To: , , , , , , , , , , CC: , , , , , , Boris Pismenny , Ben Ben-Ishay , Or Gerlitz , Yoray Zack Subject: [PATCH v5 net-next 20/36] Documentation: add ULP DDP offload documentation Date: Thu, 22 Jul 2021 14:03:09 +0300 Message-ID: <20210722110325.371-21-borisp@nvidia.com> X-Mailer: git-send-email 2.24.1 In-Reply-To: <20210722110325.371-1-borisp@nvidia.com> References: <20210722110325.371-1-borisp@nvidia.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: cd700ee0-36d1-4f36-f8f1-08d94d00ab29 X-MS-TrafficTypeDiagnostic: BN8PR12MB3251: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:10000; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: Xj6GLDDfQ0rcpXJwvdONXcZdnAKg/IHggwCzJtzQzY2KTfiKCEaU2kUpSqvDO0WE5qytMs5ymiFJbplsKgXD2kseP4NCuO5BYCXdONy6BH/vuiZtSk8oeUcCmt9QhQqzu8YFD+sRoJrziU8gz3ar3iDQ4w/fu/WbDggdfxTtMByampn6tYALanDoGh1pl0IBsIfLAgMUwX97+pIplwotugou+O0CGWQX4XkDlHe4a5UGNIkA8Fwlku+EUWWavu2kpZ+OgXuNgaEUNOQw6ZyB9rB6rv1ZIF4NDlzcIhk/TIouLq2NVYMF/uE4WRol6nTw11giZQuiTVlcvt6ZHul2VNESyK09FLfMd5lgvs93W70OJ2qG1o58cTbmyJm8AyQL43ghW/REV+zpWrq9ozDQxlfZW+JnLJDmdZSnZTqTfijWtAHsygLfQHhlx5q+CYtv0IUwgCm+DSOAoVWvddRcCJTuu/aQunSJ5f+2kZeGDjZqK2XYdS51mjcxsx7Cs3aP24KbQTQxzI4WJvsVGdOIJ0LlOGzZfdJCUnZc9KNholfqhOGAo9hMAwHaFoFosyx0h86Ul2ih6V6JiS24Xqb4489fyzhhhv3oZOo+RV8FRMyLKBoAyM/T9RgZYftajnJtvNDWuZqqdy/JmmiRakIa5Tlxm5ANvM7FG882vPYXOMqhVKTnET1ISKgkUXUGwU/fjqD82uKSFB6WHZEtm7EbqG/PEoheO0j64GWn7BBXCEQ= X-Forefront-Antispam-Report: CIP:216.228.112.34; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:schybrid03.nvidia.com; CAT:NONE; SFS:(4636009)(136003)(39860400002)(346002)(376002)(396003)(36840700001)(46966006)(83380400001)(1076003)(5660300002)(82740400003)(107886003)(186003)(316002)(7636003)(356005)(8936002)(26005)(82310400003)(6666004)(426003)(921005)(30864003)(7416002)(336012)(2906002)(47076005)(70586007)(86362001)(36860700001)(8676002)(70206006)(478600001)(7696005)(2616005)(110136005)(36756003)(4326008)(54906003)(36906005); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Jul 2021 11:05:51.4420 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: cd700ee0-36d1-4f36-f8f1-08d94d00ab29 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.112.34]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BN8NAM11FT065.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN8PR12MB3251 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Boris Pismenny Signed-off-by: Boris Pismenny Signed-off-by: Ben Ben-Ishay Signed-off-by: Or Gerlitz Signed-off-by: Yoray Zack --- Documentation/networking/index.rst | 1 + Documentation/networking/ulp-ddp-offload.rst | 415 +++++++++++++++++++ 2 files changed, 416 insertions(+) create mode 100644 Documentation/networking/ulp-ddp-offload.rst diff --git a/Documentation/networking/index.rst b/Documentation/networking/index.rst index e9ce55992aa9..87c08683e006 100644 --- a/Documentation/networking/index.rst +++ b/Documentation/networking/index.rst @@ -100,6 +100,7 @@ Contents: sysfs-tagging tc-actions-env-rules tcp-thin + ulp-ddp-offload team timestamping tipc diff --git a/Documentation/networking/ulp-ddp-offload.rst b/Documentation/networking/ulp-ddp-offload.rst new file mode 100644 index 000000000000..a7f1584defa2 --- /dev/null +++ b/Documentation/networking/ulp-ddp-offload.rst @@ -0,0 +1,415 @@ +.. SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause) + +================================= +ULP direct data placement offload +================================= + +Overview +======== + +The Linux kernel ULP direct data placement (DDP) offload infrastructure +provides tagged request-response protocols, such as NVMe-TCP, the ability to +place response data directly in pre-registered buffers according to header +tags. DDP is particularly useful for data-intensive pipelined protocols whose +responses may be reordered. + +For example, in NVMe-TCP numerous read requests are sent together and each +request is tagged using the PDU header CID field. Receiving servers process +requests as fast as possible and sometimes responses for smaller requests +bypasses responses to larger requests, i.e., read 4KB bypasses read 1GB. +Thereafter, clients correlate responses to requests using PDU header CID tags. +The processing of each response requires copying data from SKBs to read +request destination buffers; The offload avoids this copy. The offload is +oblivious to destination buffers which can reside either in userspace +(O_DIRECT) or in kernel pagecache. + +Request TCP byte-stream: + +.. parsed-literal:: + + +---------------+-------+---------------+-------+---------------+-------+ + | PDU hdr CID=1 | Req 1 | PDU hdr CID=2 | Req 2 | PDU hdr CID=3 | Req 3 | + +---------------+-------+---------------+-------+---------------+-------+ + +Response TCP byte-stream: + +.. parsed-literal:: + + +---------------+--------+---------------+--------+---------------+--------+ + | PDU hdr CID=2 | Resp 2 | PDU hdr CID=3 | Resp 3 | PDU hdr CID=1 | Resp 1 | + +---------------+--------+---------------+--------+---------------+--------+ + +The driver builds SKB page fragments that point destination buffers. +Consequently, SKBs represent the original data on the wire, which enables +*transparent* inter-operation with the network stack.To avoid copies between +SKBs and destination buffers,the layer-5 protocol (L5P) will check +``if (src == dst)`` for SKB page fragments,success indicates that data is +already placed there by NIC hardware and copy should be skipped. + +In addition, L5P might have DDGST that responsible for ensure no-error over the +network. If not offloded, ULP DDP might be not efiant as L5P will need to go +over the data and calculate it by himself, redundet DDP copy skip. +ULP DDP have support for Rx/Tx DDGST offload. On the recived side the NIC will +verify DDGST for recived pdus and update SKB->ddp_crc bit if so. +If all SKB constructing L5P pdu have ddp_crc on, L5P will skip on calculating +and verify the DDGST for the correspond pdu. On the Tx side, the NIC will be +responsible for calculating and fill the DDGST fields in the sent pdus. + +Offloading does require NIC hardware to track L5P protocol framing, similarly +to RX TLS offload (see documentation at +:ref:`Documentation/networking/tls-offload.rst `). NIC hardware +will parse PDU headers extract fields such as operation type, length, ,tag +identifier, etc. and offload only segments that correspond to tags registered +with the NIC, see the :ref:`buf_reg` section. + +Device configuration +==================== + +During driver initialization the device sets the ``NETIF_F_HW_ULP_DDP`` and +feature and installs its +:c:type:`struct ulp_ddp_ops ` +pointer in the :c:member:`ulp_ddp_ops` member of the +:c:type:`struct net_device `. + +Later, after the L5P completes its handshake offload is installed on the socket. +If offload installation fails, then the connection is handled by software as if +offload was not attempted. Offload installation should configure + +To request offload for a socket `sk`, the L5P calls :c:member:`ulp_ddp_sk_add`: + +.. code-block:: c + + int (*ulp_ddp_sk_add)(struct net_device *netdev, + struct sock *sk, + struct ulp_ddp_config *config); + +The function return 0 for success. In case of failure, L5P software should +fallback to normal non-offloaded operation. The `config` parameter indicates +the L5P type and any metadata relevant for that protocol. For example, in +NVMe-TCP the following config is used: + +.. code-block:: c + + /** + * struct nvme_tcp_ddp_config - nvme tcp ddp configuration for an IO queue + * + * @pfv: pdu version (e.g., NVME_TCP_PFV_1_0) + * @cpda: controller pdu data alignment (dwords, 0's based) + * @dgst: digest types enabled. + * The netdev will offload crc if ddp_crc is supported. + * @queue_size: number of nvme-tcp IO queue elements + * @queue_id: queue identifier + * @cpu_io: cpu core running the IO thread for this queue + */ + struct nvme_tcp_ddp_config { + struct ulp_ddp_config cfg; + + u16 pfv; + u8 cpda; + u8 dgst; + int queue_size; + int queue_id; + int io_cpu; + }; + +When offload is not needed anymore, e.g., the socket is being released, the L5P +calls :c:member:`ulp_ddp_sk_del` to release device contexts: + +.. code-block:: c + + void (*ulp_ddp_sk_del)(struct net_device *netdev, + struct sock *sk); + +TX +-- + +To request Tx offload for a socket `sk`, the L5P calls +:c:member:`ulp_ddp_int_tx_offload`: +.. code-block:: c + int ulp_ddp_init_tx_offload(struct sock *sk); + +When Tx offload is not needed anymore, e.g., the socket is being released, +the L5P calls :c:member:`ulp_ddp_release_tx_offload` to release device +contexts: + +.. code-block:: c + void ulp_ddp_release_tx_offload(struct sock *sk); + +Normal operation +================ + +At the very least, the device maintains the following state for each connection: + + * 5-tuple + * expected TCP sequence number + * mapping between tags and corresponding buffers + * current offset within PDU, PDU length, current PDU tag + +NICs should not assume any correlation between PDUs and TCP packets. Assuming +that TCP packets arrive in-order, offload will place PDU payload directly +inside corresponding registered buffers. No packets are to be delayed by NIC +offload. If offload is not possible, than the packet is to be passed as-is to +software. To perform offload on incoming packets without buffering packets in +the NIC, the NIC stores some inter-packet state, such as partial PDU headers. + +RX data-path +------------ + +After the device validates TCP checksums, it can perform DDP offload. The +packet is steered to the DDP offload context according to the 5-tuple. +Thereafter, the expected TCP sequence number is checked against the packet's +TCP sequence number. If there's a match, then offload is performed: PDU payload +is DMA written to corresponding destination buffer according to the PDU header +tag. The data should be DMAed only once, and the NIC receive ring will only +store the remaining TCP and PDU headers. + +We remark that a single TCP packet may have numerous PDUs embedded inside. NICs +can choose to offload one or more of these PDUs according to various +trade-offs. Possibly, offloading such small PDUs is of little value, and it is +better to leave it to software. + +Upon receiving a DDP offloaded packet, the driver reconstructs the original SKB +using page frags, while pointing to the destination buffers whenever possible. +This method enables seamless integration with the network stack, which can +inspect and modify packet fields transparently to the offload. + +TX data-path +------------ + +In DDGST Tx offload the DDGST calculation isn't performed in the ULP (L5P). +Instead packets reach a device driver, the driver will mark the packets +for DDGST offload based on the socket the packet is attached to, +and send them to the device for DDGST calculation and transmission. + +Both the device and the driver maintain expected TCP sequence numbers +due to the possibility of retransmissions and the lack of software fallback +once the packet reaches the device. +For segments passed in order, the driver marks the packets with +a connection identifier and hands them to the device. +The device identifies the packet as requiring DDGST offload handling and +confirms the sequence number matches its expectation. The device performs +DDGST calculation of the pdu data. +It replaces the PDU DDGST and TCP checksum with correct values. + +.. _buf_reg: + +Destination buffer registration +------------------------------- + +To register the mapping between tags and destination buffers for a socket +`sk`, the L5P calls :c:member:`ulp_ddp_setup` of :c:type:`struct ulp_ddp_ops +`: + +.. code-block:: c + + int (*ulp_ddp_setup)(struct net_device *netdev, + struct sock *sk, + struct ulp_ddp_io *io); + + +The `io` provides the buffer via scatter-gather list (`sg_table`) and +corresponding tag (`command_id`): + +.. code-block:: c + /** + * struct ulp_ddp_io - tcp ddp configuration for an IO request. + * + * @command_id: identifier on the wire associated with these buffers + * @nents: number of entries in the sg_table + * @sg_table: describing the buffers for this IO request + * @first_sgl: first SGL in sg_table + */ + struct ulp_ddp_io { + u32 command_id; + int nents; + struct sg_table sg_table; + struct scatterlist first_sgl[SG_CHUNK_SIZE]; + }; + +After the buffers have been consumed by the L5P, to release the NIC mapping of +buffers the L5P calls :c:member:`ulp_ddp_teardown` of :c:type:`struct +ulp_ddp_ops `: + +.. code-block:: c + + int (*ulp_ddp_teardown)(struct net_device *netdev, + struct sock *sk, + struct ulp_ddp_io *io, + void *ddp_ctx); + +`ulp_ddp_teardown` receives the same `io` context and an additional opaque +`ddp_ctx` that is used for asynchronous teardown, see the :ref:`async_release` +section. + +.. _async_release: + +Asynchronous teardown +--------------------- + +To teardown the association between tags and buffers and allow tag reuse NIC HW +is called by the NIC driver during `ulp_ddp_teardown`. This operation may be +performed either synchronously or asynchronously. In asynchronous teardown, +`ulp_ddp_teardown` returns immediately without unmapping NIC HW buffers. Later, +when the unmapping completes by NIC HW, the NIC driver will call up to L5P +using :c:member:`ddp_teardown_done` of :c:type:`struct ulp_ddp_ulp_ops`: + +.. code-block:: c + + void (*ddp_teardown_done)(void *ddp_ctx); + +The `ddp_ctx` parameter passed in `ddp_teardown_done` is the same on provided +in `ulp_ddp_teardown` and it is used to carry some context about the buffers +and tags that are released. + +Resync handling +=============== + +RX +-- +In presence of packet drops or network packet reordering, the device may lose +synchronization between the TCP stream and the L5P framing, and require a +resync with the kernel's TCP stack. When the device is out of sync, no offload +takes place, and packets are passed as-is to software. (resync is very similar +to TLS offload (see documentation at +:ref:`Documentation/networking/tls-offload.rst `) + +If only packets with L5P data are lost or reordered, then resynchronization may +be avoided by NIC HW that keeps tracking PDU headers. If, however, PDU headers +are reordered, then resynchronization is necessary. + +To resynchronize hardware during traffic, we use a handshake between hardware +and software. The NIC HW searches for a sequence of bytes that identifies L5P +headers (i.e., magic pattern). For example, in NVMe-TCP, the PDU operation +type can be used for this purpose. Using the PDU header length field, the NIC +HW will continue to find and match magic patterns in subsequent PDU headers. If +the pattern is missing in an expected position, then searching for the pattern +starts anew. + +The NIC will not resume offload when the magic pattern is first identified. +Instead, it will request L5P software to confirm that indeed this is a PDU +header. To request confirmation the NIC driver calls up to L5P using +:c:member:`*resync_request` of :c:type:`struct ulp_ddp_ulp_ops`: + +.. code-block:: c + + bool (*resync_request)(struct sock *sk, u32 seq, u32 flags); + +The `seq` field contains the TCP sequence of the last byte in the PDU header. +L5P software will respond to this request after observing the packet containing +TCP sequence `seq` in-order. If the PDU header is indeed there, then L5P +software calls the NIC driver using the :c:member:`ulp_ddp_resync` function of +the :c:type:`struct ulp_ddp_ops ` inside the :c:type:`struct +net_device ` while passing the same `seq` to confirm it is a PDU +header. + +.. code-block:: c + + void (*ulp_ddp_resync)(struct net_device *netdev, + struct sock *sk, u32 seq); + + +TX +-- + +Segments transmitted from an offloaded socket can get out of sync +in similar ways to the receive side-retransmissions - local drops +are possible, though network reorders are not. There is currently +one mechanism for dealing with out of order segments. + +Offload state rebuilding +~~~~~~~~~~~~~~~~~~~~~~~ + +Whenever an out of order segment is transmitted the driver provides +the device with enough information to perform DDGST offload. +This means most likely that the part of the pdu preceding the current +segment has to be passed to the device as part of the packet context, +together with its TCP sequence number. The device can then initialize its +offload state, process and discard the preceding data (to be able to insert +the DDGST value) and move onto handling the actual packet. +For doing so, L5P should map PDU and TCP sequnce number using + +Depending on the implementation the driver can either ask for a continuation +with the offload state and the new sequence number (next expected segment is +the one after the out of order one), or continue with the previous stream +state - assuming that the out of order segment was just a retransmission. +The former is simpler, and does not require retransmission detection therefore +it is the recommended method until such time it is proven inefficient. + +For doing so, L5P should map PDU and TCP sequnce number using: + +.. code-block:: c + int ulp_ddp_map_pdu_info(struct sock *sk, u32 start_seq, void *hdr, + u32 hdr_len, u32 data_len, struct request *req); + void ulp_ddp_close_pdu_info(struct sock *sk); + +While the driver can recived pdu information from ulp by calling: +.. code-block:: c + struct ulp_ddp_pdu_info *ulp_ddp_get_pdu_info(struct sock *sk, u32 seq); + +Statistics +========== + +Per L5P protocol, the following NIC driver must report statistics for the above +netdevice operations and packets processed by offload. For example, NVMe-TCP +offload reports: + + * ``rx_nvmeotcp_queue_init`` - number of NVMe-TCP Rx offload contexts created. + * ``rx_nvmeotcp_queue_teardown`` - number of NVMe-TCP Rx offload contexts + destroyed. + * ``rx_nvmeotcp_ddp_setup`` - number of DDP buffers mapped. + * ``rx_nvmeotcp_ddp_setup_fail`` - number of DDP buffers mapping that failed. + * ``rx_nvmeoulp_ddp_teardown`` - number of DDP buffers unmapped. + * ``rx_nvmeotcp_drop`` - number of packets dropped in the driver due to fatal + errors. + * ``rx_nvmeotcp_resync`` - number of packets with resync requests. + * ``rx_nvmeotcp_offload_packets`` - number of packets that used offload. + * ``rx_nvmeotcp_offload_bytes`` - number of bytes placed in DDP buffers. + * ``tx_nvmeotcp_offload_packets`` - number of Tx packets that used + DDGST offload. + * ``tx_nvmeotcp_offload_bytes`` - number of Tx bytes that used + DDGST offload. + * ``tx_nvmeotcp_ooo`` - number of TX Out of order packets. + * ``tx_nvmeotcp_dump_packets`` - number of Dump packets sent to the NIC for + sync on OOO context. + * ``tx_nvmeotcp_dump_bytes`` - number of Dump bytes sent to the NIC for + sync on OOO context. + * ``tx_nvmeotcp_resync`` - number of resync opertion due to out of order + packets + * ``tx_nvmeotcp_ctx`` - number of NVMe-TCP Tx offload contexts created. + * ``tx_nvmeotcp_resync_fail``: number of resync opertion that failed. + * ``tx_nvmeotcp_no_need_offload`` - number of OOO packet that not needed + offlaod + * ``tx_nvmeotcp_no_pdu_info`` - number of OOO packet that dontt have + pdu-info. + +NIC requirements +================ + +NIC hardware should meet the following requirements to provide this offload: + + * Offload must never buffer TCP packets. + * Offload must never modify TCP packet headers. + * Offload must never reorder TCP packets within a flow. + * Offload must never drop TCP packets. + * Offload must not depend on any TCP fields beyond the + 5-tuple and TCP sequence number. + +Error handling +============== + +TX +-- + +Packets may be redirected or rerouted by the stack to a different +device than the selected ULP DDP offload device. The stack will handle +such condition using the :c:func:`sk_validate_xmit_skb` helper +(ULP DDP code installs :c:func:`ulp_ddp_validate_xmit_skb` at this hook). +Offload maintains information about all pdu until the data is fully +acknowledged, so if skbs reach the wrong device they can be handled +by software fallback. + +Any ULP DDP device handling error on the transmission side must result +in the packet being dropped. For example if a packet got out of order +due to a bug in the stack or the device, reached the device and can't +be DDGST ofloaded, such packet must be dropped. + From patchwork Thu Jul 22 11:03:10 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Boris Pismenny X-Patchwork-Id: 485102 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-20.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER, INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 35BE4C6377D for ; Thu, 22 Jul 2021 11:06:03 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 155C26120D for ; Thu, 22 Jul 2021 11:06:03 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231805AbhGVKZ0 (ORCPT ); Thu, 22 Jul 2021 06:25:26 -0400 Received: from mail-co1nam11on2042.outbound.protection.outlook.com ([40.107.220.42]:8033 "EHLO NAM11-CO1-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S231698AbhGVKZW (ORCPT ); Thu, 22 Jul 2021 06:25:22 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=g4xqKVGkCcwxU5WBjOM9XI23AwTz40088xiALBy3EsEdzpsvUUIsxQu6hj3UYQYmDUUcwsSpjVX/rRz/vo/9bjSw0Wku+gi3TxsskBm1KhLMUZMxEM3LxeC0OmKi3wymiNlreBUlBc5r6z84BYPV0ygtQmn6vTOkEXZPUq0seK4tLuQ2sb945ZzXovawZIpguGMcZRHEMpBEpHotwYyX3MEtwDk1w8MzgMtiQwYnts+ZWQ7DnDSoa4wRrRHjJiOp0V+irh4qHa2zc3OdfCvhqC5hN1WdKh4vVFJEEuVyJo4ASNyoqQMq3wOkYD+t/BQsMe88fjDFBD4FFSnNjh/hEQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=pxpfYeCcgtIzhDJ4hiPapD/f+pU5+K76rALugvzYsjM=; b=VM9EV+/Stgo51gV4e88qfQNFJQeSuBfENiDZTqU3zfZTzBuEQ1q8BJhbVn5Wt0cQj5KbLDOoeF9NIBhmXnKgK6Fpmf/5dBSfeenpJxij/VWdqzaNn3G5brejByDrQB6sfyXvp9jh7HEdcgkATlT7WtKhfgLWX5ZeRcmw7vdiN6KFtW3eJEF5zWHZHHOf9VF6gr9siSO38xqDfX635A3kzkqZC6t1e6fcqu1RDCRQ21Ihflkr8DYwHWvASew/HOCF41WDB/Tp/ln8hLQNUpVuo65tCzhMX6S27BazWlB7gIsYoZ7TgOq5hBqokbqQ8+9JTZsUOeW8s63uiPlvqGUPSg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.34) smtp.rcpttodomain=marvell.com smtp.mailfrom=nvidia.com; dmarc=pass (p=quarantine sp=none pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=pxpfYeCcgtIzhDJ4hiPapD/f+pU5+K76rALugvzYsjM=; b=giBnFJwj0kCSoMwaQcwPX94QYDLrYToFp6vs8aw8U018Q0LYIXjVH1hJR0TDffsKfh7MOjuA/l4KmtJzthXwatWpH7jFWxu2EnNFpYldvkP/uFby9eS1Mw1Uv1P2ioytdFZETrIF5YhR8/3jhB41xndlNOco7BuaLKSJzntQQZVC0a2qIv9xFSOuhMEWn1/EipqDEz/KqStqzJNpWV89SyYugrMlAFWuR3Eey+AiexvjaqnauGuW3gbsVyakL+gEZYNS3dUYY9N+cSethoy2sHz/aUQvM5d7BN7r0c+SIVzE4c6Llns5zUJ07UTVysXfSqpTfMSCNFGLM0UeT1N5bg== Received: from BN9PR03CA0676.namprd03.prod.outlook.com (2603:10b6:408:10e::21) by BL0PR12MB2513.namprd12.prod.outlook.com (2603:10b6:207:4a::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4352.26; Thu, 22 Jul 2021 11:05:56 +0000 Received: from BN8NAM11FT058.eop-nam11.prod.protection.outlook.com (2603:10b6:408:10e:cafe::45) by BN9PR03CA0676.outlook.office365.com (2603:10b6:408:10e::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4331.23 via Frontend Transport; Thu, 22 Jul 2021 11:05:56 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.34) smtp.mailfrom=nvidia.com; marvell.com; dkim=none (message not signed) header.d=none; marvell.com; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.34 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.34; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.34) by BN8NAM11FT058.mail.protection.outlook.com (10.13.177.58) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4352.24 via Frontend Transport; Thu, 22 Jul 2021 11:05:55 +0000 Received: from HQMAIL107.nvidia.com (172.20.187.13) by HQMAIL107.nvidia.com (172.20.187.13) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Thu, 22 Jul 2021 11:05:55 +0000 Received: from vdi.nvidia.com (172.20.187.5) by mail.nvidia.com (172.20.187.13) with Microsoft SMTP Server id 15.0.1497.2 via Frontend Transport; Thu, 22 Jul 2021 11:05:50 +0000 From: Boris Pismenny To: , , , , , , , , , , CC: , , , , , Subject: [PATCH v5 net-next 21/36] net: drop ULP DDP HW offload feature if no CSUM offload feature Date: Thu, 22 Jul 2021 14:03:10 +0300 Message-ID: <20210722110325.371-22-borisp@nvidia.com> X-Mailer: git-send-email 2.24.1 In-Reply-To: <20210722110325.371-1-borisp@nvidia.com> References: <20210722110325.371-1-borisp@nvidia.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 2b35ebbd-e5d4-48e0-977b-08d94d00adc8 X-MS-TrafficTypeDiagnostic: BL0PR12MB2513: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:346; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: +lkUdXrnn4LrvhrbyrrtJo2hafQ1L3flwCba5g/9i/vMNJvVty1DpoUqdiyKNAZX/Ck78TxhYR1e5/odcUZtsVjTEvkZIEeuaxi6XHT5+h+RlqfmjT4wd+AuHq07YhGsKhCNg7xsZVLuOIt8Vt6wbGWF/5GnNpv6OA0Ao5gJSIOSQFpFZCcXUvO1xycRXH3cgRAXZyfJp9+R/wxbwVRdto8dzEWQ4xUk/WvSTfB0/G0x0j3X3tAywZ0woYtzNx4oyoY7/OWXPtdKmn9HXsw4Xoyx5JIChcxU/ZwB59nnWvp1iQZWi9mEvnBC/zhEPHTiJTWSHNy7qjy97Gv0YZ2wHhbVd918BixvebtLW11AhyPN60j/GAsvqBY+tVrNbfeYttSe5wlt7kVX3ivn7uVi3MRLhbU8AQRds9XOchJ9fa35bxq0LtozONZ4ypdojIiYsU6CBNmdp5iOsSxp8qtkyeQDGMCb4l05lxJBluudWFTA5PxYrqdGXap42DAh98P2H7Z9RELrXb8WrFMoqkXmW0ThrtPu1jx6oDgT9fK1gOuhMVeSnfmLhTuwtYXvvAdmv3IwRAyB4hhVoAL13Ojg4c7ihUvDox1kSKto6IRyeOV2vXCLwIm2cv52828aqNpQhsWGs7JESWQdvS6IUUzignD5O+1bqD1ASdOUsyACoSrdEhEK6ZtJGt1MAWAZ4TEXhTcrM1Rn4ndrPeyo82V0ySXxi7XsF18UqnxhnHLxdC8= X-Forefront-Antispam-Report: CIP:216.228.112.34; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:schybrid03.nvidia.com; CAT:NONE; SFS:(4636009)(376002)(396003)(39860400002)(346002)(136003)(46966006)(36840700001)(8936002)(186003)(4744005)(4326008)(336012)(2906002)(1076003)(82310400003)(82740400003)(7636003)(426003)(36756003)(7696005)(2616005)(107886003)(26005)(36860700001)(356005)(478600001)(8676002)(6666004)(110136005)(54906003)(5660300002)(36906005)(316002)(70206006)(86362001)(7416002)(70586007)(921005)(47076005); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Jul 2021 11:05:55.8140 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 2b35ebbd-e5d4-48e0-977b-08d94d00adc8 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.112.34]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BN8NAM11FT058.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL0PR12MB2513 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Yoray Zack Signed-off-by: Yoray Zack --- net/core/dev.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/net/core/dev.c b/net/core/dev.c index c253c2aafe97..1c8fcee2e9f9 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -9927,6 +9927,8 @@ static netdev_features_t netdev_fix_features(struct net_device *dev, if (!ip_csum && !hw_csum) { netdev_dbg(dev, "Dropping TLS TX HW offload feature since no CSUM feature.\n"); features &= ~NETIF_F_HW_TLS_TX; + netdev_dbg(dev, "Dropping ULP DDP HW offload feature since no CSUM feature.\n"); + features &= ~NETIF_F_HW_ULP_DDP; } } From patchwork Thu Jul 22 11:03:11 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Boris Pismenny X-Patchwork-Id: 484315 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-20.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER, INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 40F09C63793 for ; Thu, 22 Jul 2021 11:06:14 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 2B1DB60725 for ; Thu, 22 Jul 2021 11:06:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231824AbhGVKZh (ORCPT ); Thu, 22 Jul 2021 06:25:37 -0400 Received: from mail-dm3nam07on2063.outbound.protection.outlook.com ([40.107.95.63]:35765 "EHLO NAM02-DM3-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S231814AbhGVKZ1 (ORCPT ); Thu, 22 Jul 2021 06:25:27 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=ZAMFwu1yFTTIpEL+mYZ1vU1PWUaq/m0TLPVlF+e7jCtCQS7a7gnhayE0/SKaWQJD1Nc7ZLjVvCpmKNx+yTQcPdJWC86/lOSLcBtb7M/tQVtFHfrCXudUbk2Pp/gm8SLBuPwvZt7+eQlpcwyd33NM2nxqlVCMRFBcRlUZYkRkuSyywtxWygaMvTEY7jvtl0Kiq1iV5mibWaviSq1SITmZxqB9B2mML+i/bhyICbiJN5lOeXeqrZzJgsWI0D1Bx9blADJyljCZHvIuCBQ86CVBQJvWlS9kXPwi3UrLe9CM9o13XqXUGDNNVauO4KA9Mvr10e6/ssRWw1eZ6Sx9AS1Tcw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=mQZoz24VPjVt90atOql9PJgkSywiPU2ZImRaihLLhoU=; b=j0sUNwUNkSWHJ7fe6w+V+n2Wzn8eVSO9PLjK5eipSreM5qOkY0ucMeay1tHcfxVVlXN7G4ibZznzVtH6drZgcl27If9KKNSEU5EWYy3bzoHDlFTaU+SNMErYQWrbA9LS7wHnh0lImC3PNtRLDZhJlPj03iJM1wg/1bZOAZfHtcQ4kqo0ccqbj0diElBd9s2Rhez6oNfphwWaW1wTm2AoOlHBfuNv45nSRFIVRAdTBwVXC5a9mevukErEPTCKCrpjvikoQi/Y5NeoIjvkLrlz2mcnXFbFSucyA37MXH6SAvbsbwfKKe+YBl0QjkUebZ8xsXpT/f++LVnCtS/QNrMRuQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.36) smtp.rcpttodomain=marvell.com smtp.mailfrom=nvidia.com; dmarc=pass (p=quarantine sp=none pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=mQZoz24VPjVt90atOql9PJgkSywiPU2ZImRaihLLhoU=; b=KpHqcfy33XUp8pnLJfVkpFlk+ASn6TqaoYHAvgeyN4PDM9WYPdAUXxmhHGwan6Lo1/zp1Prd+FFfTZlmmGnVrAovhgDLIKdBc0HP/KWNZBo57T8frK80C+/Cgaq/RxkBHDIphmBETF1GqM7PEH/ichgzikVpJkFXyf8lHSKHGHiOtdwAN9WX1uI/wXTkdOG5ad2vHqYrntV/B3vBsPZOGLVTp45A/gHBAj7H/j8m/nVEfgLB8tGnjxgs7eDJrfThjYcn5komqp/w2tByp6CMaulFV3wwXCXnroXCf0p13nYVB/62xPhoxWQd4VuXZnczY4uiIs72OVNbWE/UXuIy1A== Received: from MWHPR07CA0012.namprd07.prod.outlook.com (2603:10b6:300:116::22) by BN8PR12MB3124.namprd12.prod.outlook.com (2603:10b6:408:41::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4331.24; Thu, 22 Jul 2021 11:06:00 +0000 Received: from CO1NAM11FT042.eop-nam11.prod.protection.outlook.com (2603:10b6:300:116:cafe::dd) by MWHPR07CA0012.outlook.office365.com (2603:10b6:300:116::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4352.25 via Frontend Transport; Thu, 22 Jul 2021 11:06:00 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.36) smtp.mailfrom=nvidia.com; marvell.com; dkim=none (message not signed) header.d=none; marvell.com; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.36 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.36; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.36) by CO1NAM11FT042.mail.protection.outlook.com (10.13.174.250) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4352.24 via Frontend Transport; Thu, 22 Jul 2021 11:06:00 +0000 Received: from HQMAIL107.nvidia.com (172.20.187.13) by HQMAIL101.nvidia.com (172.20.187.10) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Thu, 22 Jul 2021 11:05:59 +0000 Received: from vdi.nvidia.com (172.20.187.5) by mail.nvidia.com (172.20.187.13) with Microsoft SMTP Server id 15.0.1497.2 via Frontend Transport; Thu, 22 Jul 2021 11:05:55 +0000 From: Boris Pismenny To: , , , , , , , , , , CC: , , , , , Subject: [PATCH v5 net-next 22/36] net: Add ulp_ddp_pdu_info struct Date: Thu, 22 Jul 2021 14:03:11 +0300 Message-ID: <20210722110325.371-23-borisp@nvidia.com> X-Mailer: git-send-email 2.24.1 In-Reply-To: <20210722110325.371-1-borisp@nvidia.com> References: <20210722110325.371-1-borisp@nvidia.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 784d4371-7476-401f-08c6-08d94d00b068 X-MS-TrafficTypeDiagnostic: BN8PR12MB3124: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:3044; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: ewm/QXjiuEFXAYG/JDLW5HcPuxkhZzkFpRMjsuHICpEoQdVHnXOWusn0uyShHhG1naPGfB0804sFeI/V6srLgteUJTnBAIBOYFLFuXJjDM51Li4h1zUCL+6gw5/Hb2iNstDVXEkvOcolB9ogGwKzukslCnKuD9QGrmuboWOdcRSRjSZEKSLUPRzO09fsx+Y+94RFCEn+Jcf1A4nL5VO5a/amY5DPzCJfvzr4EfavunE1yOZkaS7dN4CQX6eISvnD8gQEMw7D3o77L9oMTKvuvMHt10sFMXnZrzyjLgRPvK0BsNT99Ohcn2d88qESYGCX4HrhUjmmwKB5069lIZp/uETxasjfs6XeQSbm7nyo1V85EA+CX4gaFMzJIWNiGj2gOL5WUnur5TxgmKkGgW76Gh2JTvhO1bSzZuZpSVF7CU08wEo68t02QC7UXHJ08Ku7ZbL2gKhvfS5ytp7cz5YJMv2gYy1eKSLVLrHqFYWSKbYeTWBXLVwY3HqYXPaNsKd6NvyFbXnJ+KUsSeYshx8gMLdXMjvtpYdWOJeD3Nl7pi7Y3ebc0Xp8qkHoLIm2rvFq/ugZKzY1IGeRFCA1UbOHOcsEEMq5PHw/klu2l47SbdpwD2/gjIjSWPgSMa4B+f2k2NikSNgP3mWvXWSpDmqvBI7jEiT5hGqr9G+pJEySvBef/bpKs4ZSUyfGHj4AxlyT5+oddtGs9aI0D97vVLmq37r6IlMc0uwypRIiRJdVm+I= X-Forefront-Antispam-Report: CIP:216.228.112.36; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:schybrid05.nvidia.com; CAT:NONE; SFS:(4636009)(396003)(346002)(136003)(39860400002)(376002)(36840700001)(46966006)(356005)(107886003)(5660300002)(921005)(82310400003)(186003)(83380400001)(70206006)(36756003)(7416002)(7636003)(70586007)(4326008)(36906005)(8676002)(316002)(2906002)(54906003)(82740400003)(336012)(8936002)(7696005)(110136005)(2616005)(426003)(6666004)(36860700001)(86362001)(1076003)(26005)(478600001)(47076005); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Jul 2021 11:06:00.3173 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 784d4371-7476-401f-08c6-08d94d00b068 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.112.36]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CO1NAM11FT042.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN8PR12MB3124 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Yoray Zack This struct is mapping between pdu's pages to TCP sequence number. The use case for this mapping is in tx offload, when the NIC needs to send a retransmitted packet. The NIC, in this case, might need the complete pdu, which is sent before (i.e for computing the CRC for this pdu). Using this mapping, the SW can send the NIC enough context to offload the packet. Signed-off-by: Yoray Zack --- include/net/ulp_ddp.h | 55 +++++++++++++- net/core/Makefile | 1 + net/core/ulp_ddp.c | 166 ++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 219 insertions(+), 3 deletions(-) create mode 100644 net/core/ulp_ddp.c diff --git a/include/net/ulp_ddp.h b/include/net/ulp_ddp.h index 1a0b464ff40b..8f48fc121c3a 100644 --- a/include/net/ulp_ddp.h +++ b/include/net/ulp_ddp.h @@ -1,4 +1,4 @@ -/* SPDX-License-Identifier: GPL-2.0 +/* SPDX-License-Identifier: GPL-3.0 * * ulp_ddp.h * Author: Boris Pismenny @@ -10,6 +10,7 @@ #include #include #include +#include /* limits returned by the offload driver, zero means don't care */ struct ulp_ddp_limits { @@ -67,6 +68,26 @@ struct ulp_ddp_io { struct scatterlist first_sgl[SG_CHUNK_SIZE]; }; +/** + * struct ulp_ddp_pdu_info - pdu info for tcp ddp crc Tx offload. + * + * @end_seq: tcp seq of the last byte in the pdu. + * @start_seq: tcp seq of the first byte in the pdu. + * @data_len: pdu data size (in bytes). + * @hdr_len: the size (in bytes) of the pdu header. + * @hdr: pdu header. + * @req: the ulp request for the original pdu. + */ +struct ulp_ddp_pdu_info { + struct list_head list; + u32 end_seq; + u32 start_seq; + u32 data_len; + u32 hdr_len; + void *hdr; + struct request *req; +}; + /* struct ulp_ddp_dev_ops - operations used by an upper layer protocol to configure ddp offload * * @ulp_ddp_limits: limit the number of scatter gather entries per IO. @@ -113,10 +134,25 @@ struct ulp_ddp_ulp_ops { /** * struct ulp_ddp_ctx - Generic ulp ddp context: device driver per queue contexts must * use this as the first member. + * + * @netdev: the coresponding netdev for this tcp ddp. + * @ddgst_len: data digest len in bytes. + * @expected_seq: indicates for next tcp seq. + * @open_info: the current pdu_info. + * @pdu_hint: hint for ulp_ddp_get_pdu_info. + * @info_list: list of the mapped pdu_infos. + * @info_lock: lock for info_list. */ struct ulp_ddp_ctx { - enum ulp_ddp_type type; - unsigned char buf[]; + enum ulp_ddp_type type; + struct net_device *netdev; + int ddgst_len; + u32 expected_seq; + struct ulp_ddp_pdu_info *open_info; + struct ulp_ddp_pdu_info *pdu_hint; + struct list_head info_list; + spinlock_t info_lock; + unsigned char buf[]; }; static inline struct ulp_ddp_ctx *ulp_ddp_get_ctx(const struct sock *sk) @@ -133,4 +169,17 @@ static inline void ulp_ddp_set_ctx(struct sock *sk, void *ctx) rcu_assign_pointer(icsk->icsk_ulp_ddp_data, ctx); } +static inline void ulp_ddp_destroy_info(struct ulp_ddp_pdu_info *info) +{ + kfree(info); +} + +void ulp_ddp_ack_handle(struct sock *sk, u32 acked_seq); +int ulp_ddp_init_tx_offload(struct sock *sk); +void ulp_ddp_release_tx_offload(struct sock *sk); +int ulp_ddp_map_pdu_info(struct sock *sk, u32 start_seq, void *hdr, + u32 hdr_len, u32 data_len, struct request *req); +void ulp_ddp_close_pdu_info(struct sock *sk); +bool ulp_ddp_need_map(struct sock *sk); +struct ulp_ddp_pdu_info *ulp_ddp_get_pdu_info(struct sock *sk, u32 seq); #endif //_ULP_DDP_H diff --git a/net/core/Makefile b/net/core/Makefile index f7f16650fe9e..b7c1618944df 100644 --- a/net/core/Makefile +++ b/net/core/Makefile @@ -14,6 +14,7 @@ obj-y += dev.o dev_addr_lists.o dst.o netevent.o \ fib_notifier.o xdp.o flow_offload.o obj-y += net-sysfs.o +obj-$(CONFIG_ULP_DDP) += ulp_ddp.o obj-$(CONFIG_PAGE_POOL) += page_pool.o obj-$(CONFIG_PROC_FS) += net-procfs.o obj-$(CONFIG_NET_PKTGEN) += pktgen.o diff --git a/net/core/ulp_ddp.c b/net/core/ulp_ddp.c new file mode 100644 index 000000000000..06ed4ad59e88 --- /dev/null +++ b/net/core/ulp_ddp.c @@ -0,0 +1,166 @@ +/* SPDX-License-Identifier: GPL-3.0 + * + * ulp_ddp.c + * Author: Yoray Zack + * Copyright (C) 2020 Mellanox Technologies. + */ +#include + +void ulp_ddp_ack_handle(struct sock *sk, u32 acked_seq) +{ + struct ulp_ddp_ctx *ctx = ulp_ddp_get_ctx(sk); + struct ulp_ddp_pdu_info *info, *temp; + unsigned long flags; + + spin_lock_irqsave(&ctx->info_lock, flags); + info = ctx->pdu_hint; + if (info && !before(acked_seq, info->end_seq)) + ctx->pdu_hint = NULL; + + list_for_each_entry_safe(info, temp, &ctx->info_list, list) { + if (before(acked_seq, info->end_seq - 1)) + break; + + list_del(&info->list); + ulp_ddp_destroy_info(info); + } + + spin_unlock_irqrestore(&ctx->info_lock, flags); +} + +static void ulp_ddp_delete_all_info(struct sock *sk) +{ + struct ulp_ddp_ctx *ctx = ulp_ddp_get_ctx(sk); + struct ulp_ddp_pdu_info *info, *temp; + unsigned long flags; + + spin_lock_irqsave(&ctx->info_lock, flags); + list_for_each_entry_safe(info, temp, &ctx->info_list, list) { + list_del(&info->list); + ulp_ddp_destroy_info(info); + } + + spin_unlock_irqrestore(&ctx->info_lock, flags); + ctx->pdu_hint = NULL; +} + +int ulp_ddp_init_tx_offload(struct sock *sk) +{ + struct ulp_ddp_ctx *ctx = ulp_ddp_get_ctx(sk); + struct ulp_ddp_pdu_info *start_marker_info; + unsigned long flags; + + start_marker_info = kzalloc(sizeof(*start_marker_info), GFP_KERNEL); + if (!start_marker_info) + return -ENOMEM; + + start_marker_info->end_seq = tcp_sk(sk)->write_seq; + start_marker_info->start_seq = tcp_sk(sk)->write_seq; + spin_lock_init(&ctx->info_lock); + INIT_LIST_HEAD(&ctx->info_list); + spin_lock_irqsave(&ctx->info_lock, flags); + list_add_tail(&start_marker_info->list, &ctx->info_list); + spin_unlock_irqrestore(&ctx->info_lock, flags); + ctx->pdu_hint = NULL; + ctx->open_info = NULL; + clean_acked_data_enable(inet_csk(sk), + &ulp_ddp_ack_handle); + return 0; +} EXPORT_SYMBOL(ulp_ddp_init_tx_offload); + +void ulp_ddp_release_tx_offload(struct sock *sk) +{ + clean_acked_data_disable(inet_csk(sk)); + ulp_ddp_delete_all_info(sk); +} EXPORT_SYMBOL(ulp_ddp_release_tx_offload); + +int ulp_ddp_map_pdu_info(struct sock *sk, u32 start_seq, void *hdr, + u32 hdr_len, u32 data_len, struct request *req) +{ + struct ulp_ddp_ctx *ctx = ulp_ddp_get_ctx(sk); + struct ulp_ddp_pdu_info *pdu_info; + u32 ddgst_len; + + pdu_info = kmalloc(sizeof(*pdu_info), GFP_KERNEL); + if (!pdu_info) + return -ENOMEM; + + ddgst_len = data_len ? ctx->ddgst_len : 0; + + pdu_info->end_seq = start_seq + hdr_len + data_len + ddgst_len; + pdu_info->start_seq = start_seq; + pdu_info->data_len = data_len; + pdu_info->hdr_len = hdr_len; + pdu_info->hdr = hdr; + pdu_info->req = req; + pdu_info->ddgst = 0; + + ctx->open_info = pdu_info; + return 0; +} EXPORT_SYMBOL(ulp_ddp_map_pdu_info); + +void ulp_ddp_close_pdu_info(struct sock *sk) +{ + struct ulp_ddp_ctx *ctx = ulp_ddp_get_ctx(sk); + struct ulp_ddp_pdu_info *pdu_info = ctx->open_info; + unsigned long flags; + + if (!pdu_info) + return; + + pdu_info->end_seq = tcp_sk(sk)->write_seq; + + spin_lock_irqsave(&ctx->info_lock, flags); + list_add_tail_rcu(&pdu_info->list, &ctx->info_list); + spin_unlock_irqrestore(&ctx->info_lock, flags); + + ctx->open_info = NULL; +} EXPORT_SYMBOL(ulp_ddp_close_pdu_info); + +bool ulp_ddp_need_map(struct sock *sk) +{ + struct ulp_ddp_ctx *ctx = ulp_ddp_get_ctx(sk); + + return !ctx->open_info; +} EXPORT_SYMBOL(ulp_ddp_need_map); + +struct ulp_ddp_pdu_info *ulp_ddp_get_pdu_info(struct sock *sk, u32 seq) +{ + struct ulp_ddp_ctx *ctx = ulp_ddp_get_ctx(sk); + struct ulp_ddp_pdu_info *info; + u32 open_start = 0; + + if (!ctx) + return NULL; + + if (ctx->open_info) { + open_start = ctx->open_info->start_seq; + if (before(open_start, seq) || seq == open_start) + return ctx->open_info; + } + + info = ctx->pdu_hint; + if (!info || before(seq, info->start_seq)) + info = list_first_entry_or_null(&ctx->info_list, + struct ulp_ddp_pdu_info, list); + + if (!info) + return NULL; + + rcu_read_lock(); + list_for_each_entry_from_rcu(info, &ctx->info_list, list) { + if (!info) + goto out; + + if (between(seq, info->start_seq, info->end_seq - 1)) { + ctx->pdu_hint = info; + goto out; + } + } + + info = NULL; +out: + rcu_read_unlock(); + return info; +} EXPORT_SYMBOL(ulp_ddp_get_pdu_info); + From patchwork Thu Jul 22 11:03:12 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Boris Pismenny X-Patchwork-Id: 485101 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-20.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER, INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C8828C63793 for ; Thu, 22 Jul 2021 11:06:19 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id AF3806121E for ; Thu, 22 Jul 2021 11:06:19 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231854AbhGVKZn (ORCPT ); Thu, 22 Jul 2021 06:25:43 -0400 Received: from mail-bn7nam10on2070.outbound.protection.outlook.com ([40.107.92.70]:14336 "EHLO NAM10-BN7-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S231820AbhGVKZb (ORCPT ); Thu, 22 Jul 2021 06:25:31 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=QnX+MsETKv22jn+k/K3vPLr+vGoHr80V1mBQXRqLI3BobQpe8dfB5khYdsAqLIe+yuVbuTfykJrtqsejNNzU/voPAsi/3G5DTyBglnws8/qOA9QT/MTIbk1Bm814Jrxs+fM2OvtRoV94FzzLl6njtrq6kqyFAClbneoygo25/eVD3ZwhUqLBZEnzbyaNmJWMD8myGXwSDHvLfu2eILUnPHgtWnEj7QPyivuoz2Rga4G9AWH7AVd7meR2CgH8jLih1vjh5oG+nnIrZooNGakx+hgV2ZKoJHPAQO8hc8QwiLgaXmAStDHQAIZne9Oh+b8YRrq5d9O6w8kX2fkc7/blJQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=MAxudF1Jfwqss5SzIz9EI8r0/Xlh8GGDSBsDbv2ESKk=; b=gJjxquzliaaWTnVozCgk9u3kes+hVARJarjFL2kXv39x5C06P5XdMHU27hpb0Ws9BnZCt4ZNocTJISwEdGe4XnF9pzPWxYYdyomtBIraiPszY5r09ZwN+qcwWD+Sd2WJqqytEQt/oRAt2eSTXXHBugxr49Et6Z3F+IDQIQjnKC+6Gct9mwibEs1JmCxEIkN/u5wn8mKaYquF0cF5DPL7AJFzcGOeSppeHRbXsiQBfE7RuwaAWlk7ndAZvTz2G+mmHNB/IfYXfHYdvKpe731QdDubykW1UhY0DcyJxso4lemMa6KvH0VET/CD+IDnhw+HAVOtQGktfoXcG3l7aL1Ejw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.36) smtp.rcpttodomain=marvell.com smtp.mailfrom=nvidia.com; dmarc=pass (p=quarantine sp=none pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=MAxudF1Jfwqss5SzIz9EI8r0/Xlh8GGDSBsDbv2ESKk=; b=TsHNHcYWUpTMdoZ0TOlAWI0jHH/vpXRN/Mb5zwuj0L3+CR7mslcSZ4eFyJmGizcePJ//JiOlhedZ9NnvtslzWOXvlQmGywgIHioXiEfJTSEB2sFBhWSJlJrm8Xuo4i+pW+0PstxQvWS37XHfG2Udgh3ytLl0cdysl7RUI6Y9x1YqXbOu8l58tsqBAUJFWzAzLwsNXWZHmgRgbxD1oIBdpV+upQULsNYgL2rDCGi0rRfxPSvMvw7fDj4Rca2zraLspaYJS/WJ4sXO0nV1Qpevnu6Y7FLZGix+qFY6fueyTu0I3baBwZaYfKPfhF1LsWxBt4YsSkadCUlzkmAVewth/A== Received: from MWHPR12CA0062.namprd12.prod.outlook.com (2603:10b6:300:103::24) by BL0PR12MB2468.namprd12.prod.outlook.com (2603:10b6:207:44::29) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4331.29; Thu, 22 Jul 2021 11:06:05 +0000 Received: from CO1NAM11FT018.eop-nam11.prod.protection.outlook.com (2603:10b6:300:103:cafe::1d) by MWHPR12CA0062.outlook.office365.com (2603:10b6:300:103::24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4352.25 via Frontend Transport; Thu, 22 Jul 2021 11:06:05 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.36) smtp.mailfrom=nvidia.com; marvell.com; dkim=none (message not signed) header.d=none; marvell.com; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.36 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.36; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.36) by CO1NAM11FT018.mail.protection.outlook.com (10.13.175.16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4352.24 via Frontend Transport; Thu, 22 Jul 2021 11:06:04 +0000 Received: from HQMAIL107.nvidia.com (172.20.187.13) by HQMAIL101.nvidia.com (172.20.187.10) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Thu, 22 Jul 2021 11:06:04 +0000 Received: from vdi.nvidia.com (172.20.187.5) by mail.nvidia.com (172.20.187.13) with Microsoft SMTP Server id 15.0.1497.2 via Frontend Transport; Thu, 22 Jul 2021 11:06:00 +0000 From: Boris Pismenny To: , , , , , , , , , , CC: , , , , , Subject: [PATCH v5 net-next 23/36] net: Add to ulp_ddp support for fallback flow Date: Thu, 22 Jul 2021 14:03:12 +0300 Message-ID: <20210722110325.371-24-borisp@nvidia.com> X-Mailer: git-send-email 2.24.1 In-Reply-To: <20210722110325.371-1-borisp@nvidia.com> References: <20210722110325.371-1-borisp@nvidia.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 4473b2b8-a685-4394-675f-08d94d00b32a X-MS-TrafficTypeDiagnostic: BL0PR12MB2468: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:8273; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: LQlVI+xvg4TqqtGi35sGzz1I74C137j84BQOjOfR5Uz4ppoeK7EDB0XtZtmhe5OjQUVhAKxhqONeqtzlibEvwQMUYAy7KsDuwRT8yDwh361ToXV8avS03RCY7bl9T3DmYX1TCTZCBFdarMu9f5dlKYZBQgnaLz37R4DCkbelqkvHrw1CeE4+9xvKTj4idi9zSZPGRKkGFyaU/bF4/90miPfATLLzKqJsVbkdDBjLOUb/SlHlf44veHsJoTxvDi9Q2KZqxT3uAZFInd9rX+AFjqSKrzzcSSqr3TY31Y4TXKOOGGkOujvpaKBDNOuyXADkkFcRo2dE95U2wr7X86PJ7/VSLnViEePhkdmkLcYX5agkrwhrFgqQ+cj8QnmaRh+jDAo1mPh4R+GoOM21rfnFFw7ngMfGQL+e7vxEUckTp/0YONYrkG69H1ZL0aCq1xsV6Klkk4q310ZEPXscZR9ygUWXTQQx7uSE9sqRLtTZwI4NjYjWqqXpsR1Sz0q1yOjdca1qeDpwEzzmvyqZ+R5Hi9Z9gwbD3zHs7x+w6JI4NMCj4kRXGbm0WV+mG5ffDpoR9dhTLmJJpVG+zIdw2AKoHDKl1XJfFQoE3SxC7jzuyUxe/YrqPPmADgRAlDt3CwhH4mLXkfza3VvAY7uedW7qkhpHXhaVOOdw000mNDqgN1+4MRTfZm7aCSR/G8N8Yq1C00IvQpYnKdbQ6ApaAXbRtIy8LXm7wE2gLG6AyeGwR5o= X-Forefront-Antispam-Report: CIP:216.228.112.36; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:schybrid05.nvidia.com; CAT:NONE; SFS:(4636009)(39860400002)(396003)(376002)(136003)(346002)(36840700001)(46966006)(8936002)(86362001)(426003)(36860700001)(36756003)(7696005)(336012)(107886003)(36906005)(921005)(316002)(478600001)(7416002)(2906002)(8676002)(54906003)(186003)(1076003)(110136005)(2616005)(70206006)(47076005)(5660300002)(82310400003)(70586007)(4326008)(6666004)(83380400001)(26005)(356005)(82740400003)(7636003); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Jul 2021 11:06:04.9735 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 4473b2b8-a685-4394-675f-08d94d00b32a X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.112.36]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CO1NAM11FT018.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL0PR12MB2468 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Yoray Zack Add ddp_ddgest_falback(), and ddp_get_pdu_info function to ulp. During DDP CRC Tx offload, the HW is responsible for calculate the crc, and therefore the SW not calculates it. If the HW changes for some reason, the SW should fallback from the offload and calculate the crc. This is checking in the ulp_ddp_validate_skb and if need fallback it do it. Signed-off-by: Yoray Zack --- include/net/ulp_ddp.h | 7 +++++ net/core/ulp_ddp.c | 69 +++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 76 insertions(+) diff --git a/include/net/ulp_ddp.h b/include/net/ulp_ddp.h index 8f48fc121c3a..40bfcfe94cef 100644 --- a/include/net/ulp_ddp.h +++ b/include/net/ulp_ddp.h @@ -77,6 +77,7 @@ struct ulp_ddp_io { * @hdr_len: the size (in bytes) of the pdu header. * @hdr: pdu header. * @req: the ulp request for the original pdu. + * @ddgst: pdu data digest. */ struct ulp_ddp_pdu_info { struct list_head list; @@ -86,6 +87,7 @@ struct ulp_ddp_pdu_info { u32 hdr_len; void *hdr; struct request *req; + __le32 ddgst; }; /* struct ulp_ddp_dev_ops - operations used by an upper layer protocol to configure ddp offload @@ -129,6 +131,8 @@ struct ulp_ddp_ulp_ops { bool (*resync_request)(struct sock *sk, u32 seq, u32 flags); /* NIC driver informs the ulp that ddp teardown is done - used for async completions*/ void (*ddp_teardown_done)(void *ddp_ctx); + /* NIC request ulp to calculate the ddgst and store it in pdu_info->ddgst */ + void (*ddp_ddgst_fallback)(struct ulp_ddp_pdu_info *pdu_info); }; /** @@ -182,4 +186,7 @@ int ulp_ddp_map_pdu_info(struct sock *sk, u32 start_seq, void *hdr, void ulp_ddp_close_pdu_info(struct sock *sk); bool ulp_ddp_need_map(struct sock *sk); struct ulp_ddp_pdu_info *ulp_ddp_get_pdu_info(struct sock *sk, u32 seq); +struct sk_buff *ulp_ddp_validate_xmit_skb(struct sock *sk, + struct net_device *dev, + struct sk_buff *skb); #endif //_ULP_DDP_H diff --git a/net/core/ulp_ddp.c b/net/core/ulp_ddp.c index 06ed4ad59e88..80366c7840a8 100644 --- a/net/core/ulp_ddp.c +++ b/net/core/ulp_ddp.c @@ -164,3 +164,72 @@ struct ulp_ddp_pdu_info *ulp_ddp_get_pdu_info(struct sock *sk, u32 seq) return info; } EXPORT_SYMBOL(ulp_ddp_get_pdu_info); +static void ulp_ddp_ddgst_recalc(const struct ulp_ddp_ulp_ops *ulp_ops, + struct ulp_ddp_pdu_info *pdu_info) +{ + if (pdu_info->ddgst) + return; + + ulp_ops->ddp_ddgst_fallback(pdu_info); +} + +static struct sk_buff *ulp_ddp_fallback_skb(struct ulp_ddp_ctx *ctx, + struct sk_buff *skb, + struct sock *sk) +{ + const struct ulp_ddp_ulp_ops *ulp_ops = inet_csk(sk)->icsk_ulp_ddp_ops; + int datalen = skb->len - (skb_transport_offset(skb) + tcp_hdrlen(skb)); + struct ulp_ddp_pdu_info *pdu_info = NULL; + int ddgst_start, ddgst_offset, ddgst_len; + u32 seq = ntohl(tcp_hdr(skb)->seq); + u32 end_skb_seq = seq + datalen; + u32 first_seq = seq; + + if (!(ulp_ops && ulp_ops->ddp_ddgst_fallback)) + return skb; + +again: + /* check if we can't use the last pdu_info + * Reasons we can't use it: + * 1. first time and then pdu_info is NULL. + * 2. seq doesn't Map to this pdu_info (out of bounds). + */ + if (!pdu_info || !between(seq, pdu_info->start_seq, pdu_info->end_seq - 1)) { + pdu_info = ulp_ddp_get_pdu_info(sk, seq); + if (!pdu_info) + return skb; + } + + ddgst_start = pdu_info->end_seq - ctx->ddgst_len; + + //check if this skb contains ddgst field + if (between(ddgst_start, seq, end_skb_seq - 1) && pdu_info->data_len) { + ulp_ddp_ddgst_recalc(ulp_ops, pdu_info); + ddgst_offset = ddgst_start - first_seq + skb_headlen(skb); + ddgst_len = min_t(int, ctx->ddgst_len, end_skb_seq - ddgst_start); + skb_store_bits(skb, ddgst_offset, &pdu_info->ddgst, ddgst_len); + } + + //check if there is more PDU's in this skb + if (between(pdu_info->end_seq, seq + 1, end_skb_seq - 1)) { + seq = pdu_info->end_seq; + goto again; + } + + return skb; +} + +struct sk_buff *ulp_ddp_validate_xmit_skb(struct sock *sk, + struct net_device *dev, + struct sk_buff *skb) +{ + struct ulp_ddp_ctx *ctx = ulp_ddp_get_ctx(sk); + + if (!ctx) + return skb; + + if (dev == ctx->netdev) + return skb; + + return ulp_ddp_fallback_skb(ctx, skb, sk); +} EXPORT_SYMBOL(ulp_ddp_validate_xmit_skb); From patchwork Thu Jul 22 11:03:13 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Boris Pismenny X-Patchwork-Id: 484314 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-20.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER, INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 98EAFC6377D for ; Thu, 22 Jul 2021 11:06:24 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 8162960725 for ; Thu, 22 Jul 2021 11:06:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231811AbhGVKZs (ORCPT ); Thu, 22 Jul 2021 06:25:48 -0400 Received: from mail-bn8nam08on2057.outbound.protection.outlook.com ([40.107.100.57]:57440 "EHLO NAM04-BN8-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S231796AbhGVKZg (ORCPT ); Thu, 22 Jul 2021 06:25:36 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=gGsr+J5/Fj7tWRU+PzSPqGJQGh5menfwCeVxytxIebOeegeYZxexTwHmsTtxl2FIyvaKrO5Zksq+bp1mJPY84povI985ux627tBbc1yVe3v2dc5Tct2t+0dq89TEFGUUFvfH7Fjo5GDlEK8VGsAd99heT1GYz0FI7TWj0LjcJ5un0DcdWzq6EUet+M5eAWg5bmdsRmcp6I+uG1nMJVWuvWC2G9xCJM1YvZKRdEONg5/yjXsgfjM63SJZ5thmC9Ki7VmJLFiq8PsOcayxU3BV8nZiI48STlhE/+SFAn9TwrnxmvTdlKyQCxphuLthswI27WE+wdzuZMp0jkStajkMkw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=DZDyYJpL1BwjLyHKPv4kjf6wRn+Kw2VKbmvpOLfLYvM=; b=cop0FimPgMZTMc9ulFrM7FYblXWPvfVsHscdxO6WkfRjbFj1MY0+1FMnnGcsdQJpxbLJWFH3Pgs9S7p122MZW9D1uveXzFcxtqly53xNHFN4xTXxUHRuvtc8uc/HHuqfVI3GNFe21E/9LZpmWPwtX3O7MEApHiDo8KkD1FH7ccD+uf2EPCRfguZx1x4ZHOkn9Tv//9941JweC4p7DjrZ5nNjr48bvO/X7XMzDoQJh+tvhO4QOEGmpqVmcy6y6sDopBv/cpIpccvhnMTIZesv2yQejnCy2tRmNlEz6vZcNHFj60DGoB8rYIwZKbvl50wQKQi2rOmtJkzjXGppK8+ZqA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.34) smtp.rcpttodomain=fb.com smtp.mailfrom=nvidia.com; dmarc=pass (p=quarantine sp=none pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=DZDyYJpL1BwjLyHKPv4kjf6wRn+Kw2VKbmvpOLfLYvM=; b=UR43ydNMduFzOaLrmy/a0X3lF38j8Al6+e8vMTmvqFvHC0pIFHb54e6lCYNKlDWiiVR6TXojiHGW2MLn+iT/heM1vJEE8T2ELCTYaK0d2xUBbCr4KVPwI50Msj/At1PdQu39mWzMAms3+mrinWs7hbM/odW13bToxRvjTLxrg9PlxF0ZwKvF477sdaOmrv5ddYFL+cKdcHT6Apc72Cex7bfuIV6ZtUKg16iWx8cxkFKqSaMnB6di0wvhcG7Adj0ngrjJAK0gyvrOakSP+DvD8xdpsE1BqF7J3FoQHAoQbMTIxDO9+yToB6PJUVr2dOxCfKA3nFb4c20nPyEzhBhaVw== Received: from BN9PR03CA0213.namprd03.prod.outlook.com (2603:10b6:408:f8::8) by CH0PR12MB5314.namprd12.prod.outlook.com (2603:10b6:610:d5::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4352.24; Thu, 22 Jul 2021 11:06:10 +0000 Received: from BN8NAM11FT042.eop-nam11.prod.protection.outlook.com (2603:10b6:408:f8:cafe::7d) by BN9PR03CA0213.outlook.office365.com (2603:10b6:408:f8::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4352.26 via Frontend Transport; Thu, 22 Jul 2021 11:06:10 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.34) smtp.mailfrom=nvidia.com; fb.com; dkim=none (message not signed) header.d=none; fb.com; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.34 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.34; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.34) by BN8NAM11FT042.mail.protection.outlook.com (10.13.177.85) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4352.24 via Frontend Transport; Thu, 22 Jul 2021 11:06:10 +0000 Received: from HQMAIL105.nvidia.com (172.20.187.12) by HQMAIL107.nvidia.com (172.20.187.13) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Thu, 22 Jul 2021 11:06:09 +0000 Received: from HQMAIL107.nvidia.com (172.20.187.13) by HQMAIL105.nvidia.com (172.20.187.12) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Thu, 22 Jul 2021 11:06:08 +0000 Received: from vdi.nvidia.com (172.20.187.5) by mail.nvidia.com (172.20.187.13) with Microsoft SMTP Server id 15.0.1497.2 via Frontend Transport; Thu, 22 Jul 2021 11:06:04 +0000 From: Boris Pismenny To: , , , , , , , , , , CC: , , , , , Subject: [PATCH v5 net-next 24/36] net: Add MSG_DDP_CRC flag Date: Thu, 22 Jul 2021 14:03:13 +0300 Message-ID: <20210722110325.371-25-borisp@nvidia.com> X-Mailer: git-send-email 2.24.1 In-Reply-To: <20210722110325.371-1-borisp@nvidia.com> References: <20210722110325.371-1-borisp@nvidia.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 5ce402c1-a1a1-40ad-44d6-08d94d00b660 X-MS-TrafficTypeDiagnostic: CH0PR12MB5314: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:1201; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: DLwMYfk0rcrhnkWQ2EjsgAPotoVfXKiEW+rC4VDv93KS8SQXxjOY7rKh9e+Ousj8VcadPKMi2Rxs/rRC6IuNA0kBwAB2d0hsWFLzsyxBgzO1DgxqX+DgTvuDj6Hs725of77AA7cEnKEPXCCowT6YCMFAdzBHj2pnJCq715DgClIrnvN/EZ5iP4PXLwTphOD/fWpXlGsAnK5jjyR/dSStlsFE6vexDfpHouLdOPr2aIVrW+8svvEvZqqqGyzWPCqXovthrUr18dGbAp9YiSwKE2ju5afgPDhMiauH1ZDIQHNtHqApN0L4iG5mUuMyfnyZoru79gAOgo4ld7B1LKSIQNXcS885mKApnDdeyL4HeKFj8WwfN9s290xc7OJ6hB6JdaurTlgBEHTh67K16/2D3GivnHQ6Cfky0f22Im14mMpTYL+3J14Y9k15uBv96fjWJ8aWhNOnn0I4K+GOZzUTgMZ3nOYcSwKNqLxYN6tmDiSKwehdxhqIR+PvaDZQLyZhT2oJ/3L7HKvppqYl3eBqno1MVzIm0/hiFm86WXmHHUE7FiMdlk8bcoNJtQMrRxNBZhrglyuJ1jTt4FBirAvjMvBQR3ZRcHZfm1VQb68749jPc/IBzuPNJIakehl6MC4BLMM1kfeOB1kV4B1bnNnbgLTnvvw3FDSSgdy/JdLxjA9LSgOCSHJ5hj680ei9AW87lgbm7+Pi5aEnhYJMKNRGzhtVkTv1HmP1AvkYvw/3MrM= X-Forefront-Antispam-Report: CIP:216.228.112.34; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:schybrid03.nvidia.com; CAT:NONE; SFS:(4636009)(376002)(396003)(39860400002)(136003)(346002)(36840700001)(46966006)(47076005)(4326008)(110136005)(478600001)(426003)(36860700001)(70206006)(70586007)(7696005)(82310400003)(2616005)(86362001)(5660300002)(2906002)(83380400001)(921005)(7416002)(316002)(82740400003)(6666004)(54906003)(36906005)(107886003)(186003)(36756003)(8676002)(1076003)(26005)(356005)(7636003)(8936002)(336012); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Jul 2021 11:06:10.2098 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 5ce402c1-a1a1-40ad-44d6-08d94d00b660 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.112.34]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BN8NAM11FT042.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH0PR12MB5314 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Yoray Zack if the msg sent with this flag, turn up skb->ddp_crc bit. Signed-off-by: Yoray Zack --- include/linux/socket.h | 1 + include/net/sock.h | 6 ++++++ net/core/sock.c | 7 +++++++ net/ipv4/tcp.c | 6 ++++++ 4 files changed, 20 insertions(+) diff --git a/include/linux/socket.h b/include/linux/socket.h index 0d8e3dcb7f88..640ec8535f43 100644 --- a/include/linux/socket.h +++ b/include/linux/socket.h @@ -310,6 +310,7 @@ struct ucred { * plain text and require encryption */ +#define MSG_DDP_CRC 0x200000 /* Skb pdu need crc offload */ #define MSG_ZEROCOPY 0x4000000 /* Use user data in kernel path */ #define MSG_FASTOPEN 0x20000000 /* Send data in TCP SYN */ #define MSG_CMSG_CLOEXEC 0x40000000 /* Set close_on_exec for file diff --git a/include/net/sock.h b/include/net/sock.h index 5fa2fd192d18..84141fdc3b80 100644 --- a/include/net/sock.h +++ b/include/net/sock.h @@ -2656,6 +2656,12 @@ static inline struct sk_buff *sk_validate_xmit_skb(struct sk_buff *skb, pr_warn_ratelimited("unencrypted skb with no associated socket - dropping\n"); kfree_skb(skb); skb = NULL; +#endif +#ifdef CONFIG_ULP_DDP + } else if (unlikely(skb->ddp_crc)) { + pr_warn_ratelimited("crc-offload skb with no associated socket - dropping\n"); + kfree_skb(skb); + skb = NULL; #endif } #endif diff --git a/net/core/sock.c b/net/core/sock.c index ba1c0f75cd45..616ffc523b5d 100644 --- a/net/core/sock.c +++ b/net/core/sock.c @@ -2160,6 +2160,13 @@ static bool can_skb_orphan_partial(const struct sk_buff *skb) */ if (skb->decrypted) return false; +#endif +#ifdef CONFIG_ULP_DDP + /* Drivers depend on in-order delivery for crc offload, + * partial orphan breaks out-of-order-OK logic. + */ + if (skb->ddp_crc) + return false; #endif return (skb->destructor == sock_wfree || (IS_ENABLED(CONFIG_INET) && skb->destructor == tcp_wfree)); diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c index d5ab5f243640..36c445ed8a30 100644 --- a/net/ipv4/tcp.c +++ b/net/ipv4/tcp.c @@ -984,6 +984,9 @@ struct sk_buff *tcp_build_frag(struct sock *sk, int size_goal, int flags, #ifdef CONFIG_TLS_DEVICE skb->decrypted = !!(flags & MSG_SENDPAGE_DECRYPTED); +#endif +#ifdef CONFIG_ULP_DDP + skb->ddp_crc = !!(flags & MSG_DDP_CRC); #endif skb_entail(sk, skb); copy = size_goal; @@ -1311,6 +1314,9 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size) if (!skb) goto wait_for_space; +#ifdef CONFIG_ULP_DDP + skb->ddp_crc = !!(flags & MSG_DDP_CRC); +#endif process_backlog++; skb->ip_summed = CHECKSUM_PARTIAL; From patchwork Thu Jul 22 11:03:14 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Boris Pismenny X-Patchwork-Id: 485100 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-20.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER, INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 30BFEC63793 for ; Thu, 22 Jul 2021 11:06:32 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 125D360725 for ; Thu, 22 Jul 2021 11:06:32 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231801AbhGVKZz (ORCPT ); Thu, 22 Jul 2021 06:25:55 -0400 Received: from mail-bn8nam12on2047.outbound.protection.outlook.com ([40.107.237.47]:64672 "EHLO NAM12-BN8-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S231648AbhGVKZo (ORCPT ); Thu, 22 Jul 2021 06:25:44 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=MoUSrgSvotWqlY2n/pHnis2GKVIdKckmWB9m6aoA4p7U2v65WokUI0+cuEXc9fbQDMtx+VCkQgCodS0e+h78lU4WoeUrPw8eBnYl7b3NxjYfhoHxuW9IM56gWNsbYEm7I5Q4+3BMnXavOwTfdgxAluJfHkrGzKzgb62xbP8HdSxPYAOJ0G7d8iBLJJPuSveipPAKqFMsPAmrwq5lozuU/up6vwZ65G75+Zzydi5w7sfeRm6VgoVAzTIGxbMhtkoBUmS75A5a0ZS3z3GkmxdPNXpFNdzGnp/LFy9t+jHCi5F/gEz6OAJkqcloe8OGmbRCkh+BUdOklC0UzE9ctU3Aeg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=6vg9NDsMpXNT2WtSCE/G2+6uynKnGET3ieyP1VjSOwI=; b=bF6bbnbC65AJJwcjsyoqJArgBPF5/BjtwAnX6deJtAqu6k64nOSsYcPvLQb27oSlg72J4P/fLMVtz549nasjEXHWXA+HINFxHLtY14S/M6RJgOEWWNsA6e8cUPPq7Wm+ay2G1FVDc0gCwAdCXP8hXcPKfCoLdk3Ckj7zY1fspCN4GRGJYdKMjUSM996QG299D6DC9b8SDbv3Dqb9OicXkFitZYiQZb8eJi/5xVb3VFfjiQkIFLqdAvtreDOcxGKF1nZah1eiZVsFhJ4/jsPGhM9ztfsO4WJt0oG4dPOuf0iR/khVEYIdrz/F5cjQpgsUZ9INT2taC4mv+kZn1l/lQw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.36) smtp.rcpttodomain=marvell.com smtp.mailfrom=nvidia.com; dmarc=pass (p=quarantine sp=none pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=6vg9NDsMpXNT2WtSCE/G2+6uynKnGET3ieyP1VjSOwI=; b=bm/Kg9CN3OIMvgNxPn6+KxWNns7wuwsgGcJO0odBM28ONLMf3Aib9WiWUvGzUueV7t2yYimeOELy2fyJT/3fXODz7OyzgHA6rC4IFVjtT9jqYrpyIkP0H8pEAk0o3w2ai+STFKSs7QD7hMKMpwRB3zQzy7bn4eeIOEjNCLlY8csMI7CrfBpVrCUmHdL61USMHY3LuDI1jMdMPolDs8JgfoDa2EWWYjhqe0Qq+LXnGpKZ+Q01bknJP9d8k0asor8+x0vgqKLOXJH8+v6UwV01y7oiMuoicOzIVt6O0cc2ba0luq0/LOe5Zpo/qQcdJQ3+NdISVCX4/tNUYJL45gOrug== Received: from MW4PR04CA0265.namprd04.prod.outlook.com (2603:10b6:303:88::30) by SN6PR12MB2750.namprd12.prod.outlook.com (2603:10b6:805:77::29) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4331.29; Thu, 22 Jul 2021 11:06:14 +0000 Received: from CO1NAM11FT065.eop-nam11.prod.protection.outlook.com (2603:10b6:303:88:cafe::57) by MW4PR04CA0265.outlook.office365.com (2603:10b6:303:88::30) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4352.26 via Frontend Transport; Thu, 22 Jul 2021 11:06:14 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.36) smtp.mailfrom=nvidia.com; marvell.com; dkim=none (message not signed) header.d=none; marvell.com; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.36 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.36; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.36) by CO1NAM11FT065.mail.protection.outlook.com (10.13.174.62) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4352.24 via Frontend Transport; Thu, 22 Jul 2021 11:06:14 +0000 Received: from HQMAIL107.nvidia.com (172.20.187.13) by HQMAIL101.nvidia.com (172.20.187.10) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Thu, 22 Jul 2021 11:06:14 +0000 Received: from vdi.nvidia.com (172.20.187.5) by mail.nvidia.com (172.20.187.13) with Microsoft SMTP Server id 15.0.1497.2 via Frontend Transport; Thu, 22 Jul 2021 11:06:09 +0000 From: Boris Pismenny To: , , , , , , , , , , CC: , , , , , Subject: [PATCH v5 net-next 25/36] nvme-tcp: TX DDGST offload Date: Thu, 22 Jul 2021 14:03:14 +0300 Message-ID: <20210722110325.371-26-borisp@nvidia.com> X-Mailer: git-send-email 2.24.1 In-Reply-To: <20210722110325.371-1-borisp@nvidia.com> References: <20210722110325.371-1-borisp@nvidia.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 4ad55c43-9e0a-4a43-1da6-08d94d00b8de X-MS-TrafficTypeDiagnostic: SN6PR12MB2750: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:256; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: g37DHSJKeP+/cMb63iyVuMHGgJhEH9Vy0riBvj563ua8p+rHiFgGiRJePKK/bRNaIxsKFwJ+vnHNGymYZDPNQFPriD+0UwnH9Kt21fOZdo51qHw7vCFSqn0oCvgvrnxXxfQIOFZ6G5kkHi/ldaUN6kOFeBaVPK+UbWwnVo4KlUSHgIxTBKPtuDWqL4bCYXrQMMD2IvyifJQl+Q1mVEfZvH9NNMy3YOshxvf+KoY/0bvjU6DMNmQ23CJWrR88iilxsgfcFYN0DUVh0sWso7Z7i7xYRieQTWpV64Df/oL9DPFluF66y6YwmMHPNsMmxVKrpB94k4MryrH4sJ+3yoY3QvhQGiMLxU3PCS6zfuArxtQMu1AwwItobjjrzvta5zRviko2cOFbOwI0WLlWyVaIEJH7OpB3v4LgIc3szzpBTbsEB2FW17mIavfc9rZbSqB3PALMB368No7AQLkY5fQEQFc+EMRKhANlm5AbhVt5gTPbQbwOEhRr5THmSz0HppkgTkNs/2Z/c1q9eWzMNc7TqIFznQ/x6uN/qp37SXmlUnyd9cl5SPCDElMogYWAww3j/bkGAKjFURbEKucamZjvrR7J1jDAZseqEbSZlj/VswIYT3QcSTmfqLbZsJS+58dbIRB/QdN74DykNUqamEFwZkcOWs8+JFnc/iP8Gjmwhn8XloOdI8fy2P24+bFww4pVu25Ec//39DIzlAdFQmjGIsDPg1yIsxAbT94np8OXwpM= X-Forefront-Antispam-Report: CIP:216.228.112.36; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:schybrid05.nvidia.com; CAT:NONE; SFS:(4636009)(39860400002)(396003)(136003)(346002)(376002)(46966006)(36840700001)(8676002)(4326008)(356005)(110136005)(82310400003)(186003)(70206006)(7636003)(1076003)(8936002)(107886003)(6666004)(2616005)(70586007)(921005)(5660300002)(36906005)(36860700001)(336012)(86362001)(83380400001)(54906003)(26005)(36756003)(82740400003)(47076005)(7696005)(316002)(2906002)(426003)(478600001)(7416002); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Jul 2021 11:06:14.5479 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 4ad55c43-9e0a-4a43-1da6-08d94d00b8de X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.112.36]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CO1NAM11FT065.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN6PR12MB2750 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Yoray Zack This patch add support for TX DDGST offload. Enable tx side of DDGST offload when supported. if supported, NVMEoTCP will: 1. Stop compute the DDGST on transmitted pdus. 2. send dummy digest (only zeros). Signed-off-by: Yoray Zack --- drivers/nvme/host/tcp.c | 33 +++++++++++++++++++++++++++------ 1 file changed, 27 insertions(+), 6 deletions(-) diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c index b338cd2d9f65..b2a4316eddce 100644 --- a/drivers/nvme/host/tcp.c +++ b/drivers/nvme/host/tcp.c @@ -70,6 +70,7 @@ enum nvme_tcp_queue_flags { NVME_TCP_Q_POLLING = 2, NVME_TCP_Q_OFF_DDP = 3, NVME_TCP_Q_OFF_DDGST_RX = 4, + NVME_TCP_Q_OFF_DDGST_TX = 5, }; enum nvme_tcp_recv_state { @@ -372,6 +373,7 @@ static int nvme_tcp_offload_socket(struct nvme_tcp_queue *queue) if (netdev->features & NETIF_F_HW_ULP_DDP) { set_bit(NVME_TCP_Q_OFF_DDP, &queue->flags); set_bit(NVME_TCP_Q_OFF_DDGST_RX, &queue->flags); + set_bit(NVME_TCP_Q_OFF_DDGST_TX, &queue->flags); } return ret; @@ -388,6 +390,7 @@ static void nvme_tcp_unoffload_socket(struct nvme_tcp_queue *queue) clear_bit(NVME_TCP_Q_OFF_DDP, &queue->flags); clear_bit(NVME_TCP_Q_OFF_DDGST_RX, &queue->flags); + clear_bit(NVME_TCP_Q_OFF_DDGST_TX, &queue->flags); netdev->ulp_ddp_ops->ulp_ddp_sk_del(netdev, queue->sock->sk); @@ -1269,6 +1272,7 @@ static void nvme_tcp_fail_request(struct nvme_tcp_request *req) static int nvme_tcp_try_send_data(struct nvme_tcp_request *req) { struct nvme_tcp_queue *queue = req->queue; + bool is_offload = test_bit(NVME_TCP_Q_OFF_DDGST_TX, &queue->flags); while (true) { struct page *page = nvme_tcp_req_cur_page(req); @@ -1277,6 +1281,9 @@ static int nvme_tcp_try_send_data(struct nvme_tcp_request *req) bool last = nvme_tcp_pdu_last_send(req, len); int ret, flags = MSG_DONTWAIT; + if (is_offload && queue->data_digest) + flags |= MSG_DDP_CRC; + if (last && !queue->data_digest && !nvme_tcp_queue_more(queue)) flags |= MSG_EOR; else @@ -1292,15 +1299,19 @@ static int nvme_tcp_try_send_data(struct nvme_tcp_request *req) if (ret <= 0) return ret; - if (queue->data_digest) + if (queue->data_digest && !is_offload) nvme_tcp_ddgst_update(queue->snd_hash, page, offset, ret); /* fully successful last write*/ if (last && ret == len) { if (queue->data_digest) { - nvme_tcp_ddgst_final(queue->snd_hash, - &req->ddgst); + if (!is_offload) + nvme_tcp_ddgst_final(queue->snd_hash, + &req->ddgst); + else + req->ddgst = 0; + req->state = NVME_TCP_SEND_DDGST; req->offset = 0; } else { @@ -1324,6 +1335,9 @@ static int nvme_tcp_try_send_cmd_pdu(struct nvme_tcp_request *req) int flags = MSG_DONTWAIT; int ret; + if (test_bit(NVME_TCP_Q_OFF_DDGST_TX, &queue->flags) && queue->data_digest) + flags |= MSG_DDP_CRC; + if (inline_data || nvme_tcp_queue_more(queue)) flags |= MSG_MORE | MSG_SENDPAGE_NOTLAST; else @@ -1357,18 +1371,21 @@ static int nvme_tcp_try_send_cmd_pdu(struct nvme_tcp_request *req) static int nvme_tcp_try_send_data_pdu(struct nvme_tcp_request *req) { + int flags = MSG_DONTWAIT | MSG_MORE | MSG_SENDPAGE_NOTLAST; struct nvme_tcp_queue *queue = req->queue; struct nvme_tcp_data_pdu *pdu = req->pdu; u8 hdgst = nvme_tcp_hdgst_len(queue); int len = sizeof(*pdu) - req->offset + hdgst; int ret; + if (test_bit(NVME_TCP_Q_OFF_DDGST_TX, &queue->flags) && queue->data_digest) + flags |= MSG_DDP_CRC; + if (queue->hdr_digest && !req->offset) nvme_tcp_hdgst(queue->snd_hash, pdu, sizeof(*pdu)); ret = kernel_sendpage(queue->sock, virt_to_page(pdu), - offset_in_page(pdu) + req->offset, len, - MSG_DONTWAIT | MSG_MORE | MSG_SENDPAGE_NOTLAST); + offset_in_page(pdu) + req->offset, len, flags); if (unlikely(ret <= 0)) return ret; @@ -1399,6 +1416,9 @@ static int nvme_tcp_try_send_ddgst(struct nvme_tcp_request *req) else msg.msg_flags |= MSG_EOR; + if (test_bit(NVME_TCP_Q_OFF_DDGST_TX, &queue->flags)) + msg.msg_flags |= MSG_DDP_CRC; + ret = kernel_sendmsg(queue->sock, &msg, &iov, 1, iov.iov_len); if (unlikely(ret <= 0)) return ret; @@ -1908,7 +1928,8 @@ static void __nvme_tcp_stop_queue(struct nvme_tcp_queue *queue) cancel_work_sync(&queue->io_work); if (test_bit(NVME_TCP_Q_OFF_DDP, &queue->flags) || - test_bit(NVME_TCP_Q_OFF_DDGST_RX, &queue->flags)) + test_bit(NVME_TCP_Q_OFF_DDGST_RX, &queue->flags) || + test_bit(NVME_TCP_Q_OFF_DDGST_TX, &queue->flags)) nvme_tcp_unoffload_socket(queue); } From patchwork Thu Jul 22 11:03:15 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Boris Pismenny X-Patchwork-Id: 484313 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-20.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER, INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3F57BC63797 for ; Thu, 22 Jul 2021 11:06:33 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 2D57A61221 for ; Thu, 22 Jul 2021 11:06:33 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231796AbhGVKZ4 (ORCPT ); Thu, 22 Jul 2021 06:25:56 -0400 Received: from mail-mw2nam10on2044.outbound.protection.outlook.com ([40.107.94.44]:45665 "EHLO NAM10-MW2-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S231787AbhGVKZq (ORCPT ); Thu, 22 Jul 2021 06:25:46 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=hnhQ9i5QlvfeWrnDKX7QOf+blAaSZlmFHZWsg/yx9MhqD2DBeoXHC97XGS4ie4ENKqc10t/ALd0izlW4dQgNrnjFxlkas97dd2Duo631nBRCCB031mGqbAD7CcrI38cOrgZhW8AviWHhhvVzcAU8xcM7WbsEFm2cCc4uZx9JMW9BoqxJ9WNRk+yad0E+myQL5jBCEiE1jAfzYbKrAzWdE9UR+jOkUXnC0afOs5uSAP668klH7B/Sl4+wMmP6WIa+z6r6L50KIQcxsiuDUaU8JvQ2VSkDth/MIToENvpXL5aJRJUuj3w2NdinJMC8XT/67VkMDGbAtPOdVumPbSwHEg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=9DXfAQaODcu21Hc+bFAT+d5pV1pV6/PALQOiipDJy6k=; b=iKRhk+x891SZTY/DbvzeeOpVe9j6kAJ3CkrHkm0kFkW2tEldUPtcCmfc4KfSI08VCjwCcFCTbjZ14HFabePa674dMiOEVvl6MuIjX0wHEj9Khxh4tBOQULPl86qAYQmEZZ7N3tVQgju4mtxPExebedNSLkpeO6ngi4TuktBD67YCAXsvxWKAEkIar62eFnP39gsMI2AcpBdSlSK7qR6FJIcDJ2ChY3kmjQy66t3UwOlBeGLjnID13NHxRNDGPzCXudAmuC49G7m7Z1+MmFzULxUEVF3Vo76hkhcuwPnBJ19TAiFqn/25USCViuzxt0Bh3fOLZQuaS75cHKjtCtU0GQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.32) smtp.rcpttodomain=marvell.com smtp.mailfrom=nvidia.com; dmarc=pass (p=quarantine sp=none pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=9DXfAQaODcu21Hc+bFAT+d5pV1pV6/PALQOiipDJy6k=; b=L9kuNVVXWEjP0mGJD0UuHK+DNFdgWXJuI3b9gaLbTaxkjlCGF/Bm/Hg7o4L/2hGsYz+2f4IwuCwx3LYMZsLG9PHR3hRE2kr1X+oxlPM9CfYiLOZYfFCbfvPBJqWhVPa9juh3C3FiJvzBGhRx1VxCviHwM6bqdFEU9pj4Vg3YSQCmmZIJ+EAdqGD/i30eM92Qp8g6CehBCd8dh1Aner1fXzPjIHfObPAncvSYGcR/hAHXDFqYHu7H2UTzwr+a9QlVmecu8vEo1AGCokL5Jkr7+ZZUzwTBtt5P42nPbxYsW6aKszNSPqEGQa3OQbu166QrnZ4+zjjPbd3AIYFhmN/yRg== Received: from DM5PR1401CA0005.namprd14.prod.outlook.com (2603:10b6:4:4a::15) by MN2PR12MB3838.namprd12.prod.outlook.com (2603:10b6:208:16c::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4352.25; Thu, 22 Jul 2021 11:06:20 +0000 Received: from DM6NAM11FT054.eop-nam11.prod.protection.outlook.com (2603:10b6:4:4a:cafe::30) by DM5PR1401CA0005.outlook.office365.com (2603:10b6:4:4a::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4352.25 via Frontend Transport; Thu, 22 Jul 2021 11:06:19 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.32) smtp.mailfrom=nvidia.com; marvell.com; dkim=none (message not signed) header.d=none; marvell.com; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.32 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.32; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.32) by DM6NAM11FT054.mail.protection.outlook.com (10.13.173.95) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4352.24 via Frontend Transport; Thu, 22 Jul 2021 11:06:19 +0000 Received: from HQMAIL107.nvidia.com (172.20.187.13) by HQMAIL109.nvidia.com (172.20.187.15) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Thu, 22 Jul 2021 04:06:18 -0700 Received: from vdi.nvidia.com (172.20.187.5) by mail.nvidia.com (172.20.187.13) with Microsoft SMTP Server id 15.0.1497.2 via Frontend Transport; Thu, 22 Jul 2021 11:06:14 +0000 From: Boris Pismenny To: , , , , , , , , , , CC: , , , , , Subject: [PATCH v5 net-next 26/36] nvme-tcp: Mapping between Tx NVMEoTCP pdu and TCP sequence Date: Thu, 22 Jul 2021 14:03:15 +0300 Message-ID: <20210722110325.371-27-borisp@nvidia.com> X-Mailer: git-send-email 2.24.1 In-Reply-To: <20210722110325.371-1-borisp@nvidia.com> References: <20210722110325.371-1-borisp@nvidia.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: dde884d6-4271-472c-008a-08d94d00bbe7 X-MS-TrafficTypeDiagnostic: MN2PR12MB3838: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:580; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: Jc5hW52iz0l4l3T3bzDIE4gxLlkHcWGxF3scS0gxjgNLbPgTwoZkNWHS3LPNBNHFHwbBnn5EtS5ilkpWjWk+63nJN35l8sbKmeaTsYw6aCr98E+WOHkcBs8ScYs8Z9eYVQ0j3bhiWNc9iR095Cp7IszLfU6nSFbsNNVQzL4FDD+p6mpNs/7Q1lSRa6KPZT/p+L/XEyFPgB0YZln07wfG/FyFusxm0iFuOdLtbYkr7wvKVSpnB7WQSGwxDWZV/F3Y/pXeEzQ0iPawZCjhMxY93FVQ5QhG3ygl12dT8mEp7l5IjfvPRtSUPPcqB6pUbbsFnfEKJFtk+UL3d7rJ6YKSa+mwKNefS3QI6HfEF+8wLw/BHWGC73v+OT1gzNfQ+wZb9S5OAy2Y6Anq2/HicmY98JDnrwHfv+Lrfb2K2uSGOpwGWN9BSerUUAaWUKWoy2ETPvgV2+eWk2ihr4gOqV63WO/OxFL5w1h0tQxp3ohAeCvvLiWpe5juE1N4NAMfR4ay0G7kSmSnDV/OGXqlsRdLph1MYX/K5pHusErO6BgF+bFiQrjtEQKQVBN73xgV2b3HnyRHgh1ydAAG9uW0TCrMoPCd8mUpJmrsvxhy/KOkmbE2tW82uE/MGyQMyghuq3dpA0uDoma4KhnX/gBy86gsEh7y0Mq9YQI0Zzkh31+Cp98sPQyT60oASErfPsVtnmRsMaXCrEuMuRfSAMLzyZcNr7FWKQjG1c2yPJf2ra6zRKc= X-Forefront-Antispam-Report: CIP:216.228.112.32; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:schybrid01.nvidia.com; CAT:NONE; SFS:(4636009)(136003)(39860400002)(396003)(376002)(346002)(46966006)(36840700001)(70206006)(2906002)(70586007)(54906003)(7416002)(336012)(5660300002)(107886003)(478600001)(8676002)(110136005)(7696005)(2616005)(8936002)(426003)(4326008)(26005)(186003)(7636003)(356005)(82740400003)(86362001)(47076005)(36756003)(83380400001)(36860700001)(921005)(82310400003)(1076003)(316002); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Jul 2021 11:06:19.6403 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: dde884d6-4271-472c-008a-08d94d00bbe7 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.112.32]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DM6NAM11FT054.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR12MB3838 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Yoray Zack This commit maintains a mapping from TCP sequence number to NVMEoTCP pdus, for DDGST tx offload using the ULP_DDP API. When send a pdu it save the req in ulp_ddp_pdu_info struct. This mapping is used: 1. When packet is retransmitted, If this packet contain NVMEoTCP DDGST, The NIC might needs all the pdu again for computing the DDGST. 2. If packet is offloaded but will not go to the offloaded netdev, Then SW will need to be able to fallback and compute the DDGST. Add founction nvme_tcp_ddgest_fallback(pdu_info): caclulate the data digest for ulp_ddp_pdu_info (requested by the netdev). Signed-off-by: Yoray Zack --- drivers/nvme/host/tcp.c | 93 ++++++++++++++++++++++++++++++++++++++++- 1 file changed, 91 insertions(+), 2 deletions(-) diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c index b2a4316eddce..e030d1baa6bb 100644 --- a/drivers/nvme/host/tcp.c +++ b/drivers/nvme/host/tcp.c @@ -154,6 +154,55 @@ static const struct blk_mq_ops nvme_tcp_mq_ops; static const struct blk_mq_ops nvme_tcp_admin_mq_ops; static int nvme_tcp_try_send(struct nvme_tcp_queue *queue); +#ifdef CONFIG_ULP_DDP +static int nvme_tcp_map_pdu_info(struct nvme_tcp_queue *queue, + size_t sent_size, + u32 pdu_len, u32 data_len) +{ + u32 start_seq = tcp_sk(queue->sock->sk)->write_seq - sent_size; + struct nvme_tcp_request *req = queue->request; + struct request *rq = blk_mq_rq_from_pdu(req); + + return ulp_ddp_map_pdu_info(queue->sock->sk, start_seq, req->pdu, + pdu_len, data_len, rq); +} + +static void nvme_tcp_close_pdu_info(struct nvme_tcp_queue *queue) +{ + if (queue->data_digest && + test_bit(NVME_TCP_Q_OFF_DDGST_TX, &queue->flags)) + ulp_ddp_close_pdu_info(queue->sock->sk); +} + +bool nvme_tcp_need_map(struct nvme_tcp_queue *queue) +{ + return queue->data_digest && + test_bit(NVME_TCP_Q_OFF_DDGST_TX, &queue->flags) + && queue->sock && queue->sock->sk + && ulp_ddp_need_map(queue->sock->sk); + +} +#else + +static int nvme_tcp_map_pdu_info(struct nvme_tcp_queue *queue, + size_t sent_size, + u32 pdu_len, u32 data_len) +{ + return 0; +} + +static void nvme_tcp_close_pdu_info(struct nvme_tcp_queue *queue) +{ +} + +bool nvme_tcp_need_map(struct nvme_tcp_queue *queue) +{ + return false; +} +#endif + + + static inline struct nvme_tcp_ctrl *to_tcp_ctrl(struct nvme_ctrl *ctrl) { return container_of(ctrl, struct nvme_tcp_ctrl, ctrl); @@ -285,11 +334,13 @@ static void nvme_tcp_ddp_ddgst_recalc(struct ahash_request *hash, #ifdef CONFIG_ULP_DDP +void nvme_tcp_ddp_ddgst_fallback(struct ulp_ddp_pdu_info *pdu_info); static bool nvme_tcp_resync_request(struct sock *sk, u32 seq, u32 flags); static void nvme_tcp_ddp_teardown_done(void *ddp_ctx); static const struct ulp_ddp_ulp_ops nvme_tcp_ddp_ulp_ops = { .resync_request = nvme_tcp_resync_request, .ddp_teardown_done = nvme_tcp_ddp_teardown_done, + .ddp_ddgst_fallback = nvme_tcp_ddp_ddgst_fallback, }; static int nvme_tcp_teardown_ddp(struct nvme_tcp_queue *queue, @@ -371,6 +422,12 @@ static int nvme_tcp_offload_socket(struct nvme_tcp_queue *queue) inet_csk(queue->sock->sk)->icsk_ulp_ddp_ops = &nvme_tcp_ddp_ulp_ops; if (netdev->features & NETIF_F_HW_ULP_DDP) { + if (ulp_ddp_init_tx_offload(queue->sock->sk)) { + netdev->ulp_ddp_ops->ulp_ddp_sk_del(netdev, queue->sock->sk); + dev_put(netdev); + return -ENOMEM; + } + set_bit(NVME_TCP_Q_OFF_DDP, &queue->flags); set_bit(NVME_TCP_Q_OFF_DDGST_RX, &queue->flags); set_bit(NVME_TCP_Q_OFF_DDGST_TX, &queue->flags); @@ -392,6 +449,9 @@ static void nvme_tcp_unoffload_socket(struct nvme_tcp_queue *queue) clear_bit(NVME_TCP_Q_OFF_DDGST_RX, &queue->flags); clear_bit(NVME_TCP_Q_OFF_DDGST_TX, &queue->flags); + if (test_bit(NVME_TCP_Q_OFF_DDGST_TX, &queue->flags)) + ulp_ddp_release_tx_offload(queue->sock->sk); + netdev->ulp_ddp_ops->ulp_ddp_sk_del(netdev, queue->sock->sk); inet_csk(queue->sock->sk)->icsk_ulp_ddp_ops = NULL; @@ -1269,6 +1329,19 @@ static void nvme_tcp_fail_request(struct nvme_tcp_request *req) nvme_tcp_end_request(blk_mq_rq_from_pdu(req), NVME_SC_HOST_PATH_ERROR); } +#ifdef CONFIG_ULP_DDP +void nvme_tcp_ddp_ddgst_fallback(struct ulp_ddp_pdu_info *pdu_info) +{ + struct request *rq = pdu_info->req; + struct nvme_tcp_request *req = blk_mq_rq_to_pdu(rq); + struct nvme_tcp_queue *queue = req->queue; + + nvme_tcp_ddp_ddgst_recalc(queue->snd_hash, rq); + nvme_tcp_ddgst_final(queue->snd_hash, &pdu_info->ddgst); +} + +#endif + static int nvme_tcp_try_send_data(struct nvme_tcp_request *req) { struct nvme_tcp_queue *queue = req->queue; @@ -1333,7 +1406,8 @@ static int nvme_tcp_try_send_cmd_pdu(struct nvme_tcp_request *req) int len = sizeof(*pdu) + hdgst - req->offset; struct request *rq = blk_mq_rq_from_pdu(req); int flags = MSG_DONTWAIT; - int ret; + int ret, check; + u32 data_len; if (test_bit(NVME_TCP_Q_OFF_DDGST_TX, &queue->flags) && queue->data_digest) flags |= MSG_DDP_CRC; @@ -1353,6 +1427,13 @@ static int nvme_tcp_try_send_cmd_pdu(struct nvme_tcp_request *req) if (unlikely(ret <= 0)) return ret; + if (nvme_tcp_need_map(queue)) { + data_len = inline_data ? req->data_len : 0; + check = nvme_tcp_map_pdu_info(queue, ret, len, data_len); + if (unlikely(check)) + return check; + } + len -= ret; if (!len) { if (inline_data) { @@ -1360,6 +1441,7 @@ static int nvme_tcp_try_send_cmd_pdu(struct nvme_tcp_request *req) if (queue->data_digest) crypto_ahash_init(queue->snd_hash); } else { + nvme_tcp_close_pdu_info(queue); nvme_tcp_done_send_req(queue); } return 1; @@ -1376,7 +1458,7 @@ static int nvme_tcp_try_send_data_pdu(struct nvme_tcp_request *req) struct nvme_tcp_data_pdu *pdu = req->pdu; u8 hdgst = nvme_tcp_hdgst_len(queue); int len = sizeof(*pdu) - req->offset + hdgst; - int ret; + int ret, check; if (test_bit(NVME_TCP_Q_OFF_DDGST_TX, &queue->flags) && queue->data_digest) flags |= MSG_DDP_CRC; @@ -1389,6 +1471,12 @@ static int nvme_tcp_try_send_data_pdu(struct nvme_tcp_request *req) if (unlikely(ret <= 0)) return ret; + if (nvme_tcp_need_map(queue)) { + check = nvme_tcp_map_pdu_info(queue, ret, len, req->data_len); + if (unlikely(check)) + return check; + } + len -= ret; if (!len) { req->state = NVME_TCP_SEND_DATA; @@ -1424,6 +1512,7 @@ static int nvme_tcp_try_send_ddgst(struct nvme_tcp_request *req) return ret; if (req->offset + ret == NVME_TCP_DIGEST_LENGTH) { + nvme_tcp_close_pdu_info(queue); nvme_tcp_done_send_req(queue); return 1; } From patchwork Thu Jul 22 11:03:16 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Boris Pismenny X-Patchwork-Id: 485099 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-20.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER, INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9DF2FC63793 for ; Thu, 22 Jul 2021 11:06:37 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 85E146120D for ; Thu, 22 Jul 2021 11:06:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231566AbhGVKZ6 (ORCPT ); Thu, 22 Jul 2021 06:25:58 -0400 Received: from mail-bn8nam12on2057.outbound.protection.outlook.com ([40.107.237.57]:64096 "EHLO NAM12-BN8-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S231802AbhGVKZv (ORCPT ); Thu, 22 Jul 2021 06:25:51 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Si8cQcfcUvqfdZWm4q/a87aWaTsi5GcClHFnP29LQvMbLGNU+nZ0xmZoLO0HbTcYyj3ur/PWQcVAJh7fDsryXZa4yVt46QVojUgUfRjArJ/FmnYzu8PY0vP/Ha5kKUochxMK5JUP4aCERXj6+4MNUVZbOGn2AjVprYy7vl5Pna2TllRO18qPWKvdo2uFxI+VTVt+Kruyogk4AdSk/c6u+2zhTpMfHgZkapFCnuxY706xmL+J00KIY/h43UIvXxsZdPyiCYFY3Xv9gRzs9ogMKCn5FzcLXQRpAXdiOl1CfTX8oQkfurgxdv/aY1n+fAL/e2eLkCOBsAZKL6rdpkpe0Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=23J7hqJpBOuG1GMTJqcmOPI6EEn8sCEsgbnQNG4beLk=; b=MdvhpPqCa8YYcvXMpGZJ3ZfddMWDB4RhVtQshtHoZYAS5D+rr7kpt2E18PLuSrmoPwIG4t5sN0We7+tvg39OWFqJixysjKc42/QYOsI0Cq1PuZDENYxceKF+URkvkRa3VrWxiEUnv4jEKBGIHx6g5+rxOudgEDzChbsVOZfmQNNTpicjSsEgAMJSg9p6zdl9ZzchL0jdtgESXBbkDNk2XBvPb4Mxzv7k1QvtyswKkrM+YsqA69/iwJAF/As45lDXu0zdf+HMFojFkbpLyic4e1FR2viMjThZOPPreDXSh+XaMSVFXHVhcuOKo5hDxK/5j1qZzDoUiCVQdEpm0PMoZQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.32) smtp.rcpttodomain=marvell.com smtp.mailfrom=nvidia.com; dmarc=pass (p=quarantine sp=none pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=23J7hqJpBOuG1GMTJqcmOPI6EEn8sCEsgbnQNG4beLk=; b=FBKlBpDS+BUw82y/wp+tE9RsWpkPc7YlaJuw9yYyIsQmV+iR2y34dUKKP6V1T++y68AAZkGKT/hBJ7RBjoNDC8MgC3Hk4Vj7zXfc3TaVFJGWomMP0LuMVEDAO9LkwT/5XwT7sGtJQzNw/4Zr2zX/sbyW6xQkvl+a4S/TpSe5yK+gM9A+cRQG5gWuWD5b6wvYlIk3luowSkIF2UiZTfTdS+GPvysMMnyGuWJebbVdd3df1yt6xp8feAi79ikOQ/jEruH2FnKvbs6KPEQqhBylVGaycV2p5Rx300YxvSN4p2Y09EQWS7qVfEt1yY4p1Wz0WU7pxOOQ+3vCSRZjg84/HA== Received: from DM6PR06CA0033.namprd06.prod.outlook.com (2603:10b6:5:120::46) by BYAPR12MB3239.namprd12.prod.outlook.com (2603:10b6:a03:137::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4331.29; Thu, 22 Jul 2021 11:06:25 +0000 Received: from DM6NAM11FT048.eop-nam11.prod.protection.outlook.com (2603:10b6:5:120:cafe::9c) by DM6PR06CA0033.outlook.office365.com (2603:10b6:5:120::46) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4352.26 via Frontend Transport; Thu, 22 Jul 2021 11:06:24 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.32) smtp.mailfrom=nvidia.com; marvell.com; dkim=none (message not signed) header.d=none; marvell.com; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.32 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.32; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.32) by DM6NAM11FT048.mail.protection.outlook.com (10.13.173.114) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4352.24 via Frontend Transport; Thu, 22 Jul 2021 11:06:24 +0000 Received: from HQMAIL107.nvidia.com (172.20.187.13) by HQMAIL109.nvidia.com (172.20.187.15) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Thu, 22 Jul 2021 04:06:24 -0700 Received: from vdi.nvidia.com (172.20.187.5) by mail.nvidia.com (172.20.187.13) with Microsoft SMTP Server id 15.0.1497.2 via Frontend Transport; Thu, 22 Jul 2021 11:06:19 +0000 From: Boris Pismenny To: , , , , , , , , , , CC: , , , , , Subject: [PATCH v5 net-next 27/36] mlx5e: make preparation in TLS code for NVMEoTCP CRC Tx offload Date: Thu, 22 Jul 2021 14:03:16 +0300 Message-ID: <20210722110325.371-28-borisp@nvidia.com> X-Mailer: git-send-email 2.24.1 In-Reply-To: <20210722110325.371-1-borisp@nvidia.com> References: <20210722110325.371-1-borisp@nvidia.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: c8487d01-3690-4ae1-70f7-08d94d00bec3 X-MS-TrafficTypeDiagnostic: BYAPR12MB3239: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:4125; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: IW8VFQ1ACzrpQQTv3agY6Jdx9PwDI+V4RL4/RPSMm49433TN4npr0VFWXUdX2t+/ij/yYKX9ve83AoIqsb7/hFY4ThUmDAH0gxz6+NNhFU+6BYokHWtYbL3nEUOLHrStqmDQz4FbRzUCNu84vUQQl4nbeGwOROhGvb1dvQEMcD9ucS5HW2pnIzlUGjOXDcylH38FUzPVe4ueyRBFQxpLH1Wo41sidOavOs4u30jRTek55JLPEw03qM9Cgvy0DJMqMHSy2AxhYXMbeHJYKmGcHBFzMNhXWFslC3MlY5T19YQ0VU0D75aw1auv+Hwn2HQhIo6oai0C+zKSTJqZadnynSGfiqfV25CHxtup51JZ/NCRaRQk2jZmGxK8nPV4mU960fwGkWXYkQrT6rZ37JEEYWfvs+M5n61PH+fSqkYF+DRjmleDhi82YRejVbIVgDkLgiyxiZqBOOmjD1ySZP/rNeYVg2GETW8U8HdVC4jEsKP3Qlet9s9LGUu3XBTRnxIiZrAvVDdpVgPhu3edpT/Cf14rhQtEpBTKJqkM94FN9GsvYg7Q1PoJl3GmUVjVyzq94rh1sUTDgueo4EZ2WutBE9JdmOuQxjDGcXNnCeutRm+JyS13ltc2GR1janX1Pc3lSyIjxod8duvbZaOUL4XOtwJkOJ1DtJ5skKmgh3pvafk8B2Y4Ov54HOP1uRuPQ79CvF2zD1+P2jFHimegHJ9K0g2ue3aOYdd3XTdKOG1Upfr2IIda7H2fPQG6rhyEsqrU X-Forefront-Antispam-Report: CIP:216.228.112.32; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:schybrid01.nvidia.com; CAT:NONE; SFS:(4636009)(46966006)(36840700001)(2616005)(426003)(36860700001)(54906003)(6666004)(83380400001)(336012)(110136005)(8936002)(36756003)(86362001)(508600001)(2906002)(316002)(4326008)(921005)(70586007)(82310400003)(107886003)(47076005)(5660300002)(7696005)(70206006)(1076003)(7636003)(356005)(8676002)(7416002)(26005)(186003)(32563001); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Jul 2021 11:06:24.4264 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: c8487d01-3690-4ae1-70f7-08d94d00bec3 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.112.32]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DM6NAM11FT048.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR12MB3239 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Yoray Zack NVMEoTCP CRC Tx offload is similar to TLS Tx offload, and uses DUMP wqe as well. To avoid duplicate functions the following changes were added: 1. Add DUMP_WQE.type field (=TLS or NVMEoTCP). 2. change in mlx5e_ktls_tx_handle_resync_dump_comp to handle also NVMEoTCP Tx DUMP WQE. Signed-off-by: Yoray Zack --- drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h | 5 +++++ .../ethernet/mellanox/mlx5/core/en_accel/ktls_tx.c | 12 ++++++++++-- 2 files changed, 15 insertions(+), 2 deletions(-) diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h b/drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h index f0190ee6e42c..c7f979dfdd69 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h +++ b/drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h @@ -77,6 +77,10 @@ u16 mlx5e_select_queue(struct net_device *dev, struct sk_buff *skb, netdev_tx_t mlx5e_xmit(struct sk_buff *skb, struct net_device *dev); bool mlx5e_poll_tx_cq(struct mlx5e_cq *cq, int napi_budget); void mlx5e_free_txqsq_descs(struct mlx5e_txqsq *sq); +enum mlx5e_dump_wqe_type { + MLX5E_DUMP_WQE_TLS, + MLX5E_DUMP_WQE_NVMEOTCP, +}; static inline bool mlx5e_wqc_has_room_for(struct mlx5_wq_cyc *wq, u16 cc, u16 pc, u16 n) @@ -140,6 +144,7 @@ struct mlx5e_tx_wqe_info { u8 num_fifo_pkts; #ifdef CONFIG_MLX5_EN_TLS struct page *resync_dump_frag_page; + enum mlx5e_dump_wqe_type type; #endif }; diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_tx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_tx.c index 9ad3459fb63a..64780d0143ec 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_tx.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_tx.c @@ -154,6 +154,7 @@ static void tx_fill_wi(struct mlx5e_txqsq *sq, .num_wqebbs = num_wqebbs, .num_bytes = num_bytes, .resync_dump_frag_page = page, + .type = MLX5E_DUMP_WQE_TLS, }; } @@ -358,8 +359,15 @@ void mlx5e_ktls_tx_handle_resync_dump_comp(struct mlx5e_txqsq *sq, mlx5e_tx_dma_unmap(sq->pdev, dma); put_page(wi->resync_dump_frag_page); - stats->tls_dump_packets++; - stats->tls_dump_bytes += wi->num_bytes; + + switch (wi->type) { + case MLX5E_DUMP_WQE_TLS: + stats->tls_dump_packets++; + stats->tls_dump_bytes += wi->num_bytes; + break; + case MLX5E_DUMP_WQE_NVMEOTCP: + break; + } } static void tx_post_fence_nop(struct mlx5e_txqsq *sq) From patchwork Thu Jul 22 11:03:17 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Boris Pismenny X-Patchwork-Id: 484312 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-20.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER, INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9B24CC6377D for ; Thu, 22 Jul 2021 11:06:39 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 850FD6120D for ; Thu, 22 Jul 2021 11:06:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231835AbhGVK0D (ORCPT ); Thu, 22 Jul 2021 06:26:03 -0400 Received: from mail-dm6nam08on2053.outbound.protection.outlook.com ([40.107.102.53]:22369 "EHLO NAM04-DM6-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S231691AbhGVKZ5 (ORCPT ); Thu, 22 Jul 2021 06:25:57 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=aXOxi2YxmS3eoFFcwMC5tDKTAC/E9hOWUHn91KR7w1A9OgA+7zTDYEPAWdscN1/VhlS/yLYvIUQSnJ9FzXOZ6JuexYCf/wR1Wo2/3Xu+u2PtF44fdH6PJfTbw5RLlhdsykysDMycD3uMO9TmH/DweaMgHZCtRxMFvflIQRM9FUFpcdk+Hv9tvc/1tqh4VV2seXMx7KJRpA8M5tYjh4wjH6Um08OSfm/6dnDDEw6jRkY2BpnvIdCJfueYva5sXRYAjVCChhhPVA2X0ODSnm+7g90D2nh1ohSl7i6DkZDy6g7ey6xLkEky1Jwx9CxJ89VZ+VPPamPT0aaySC6Cws28MQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=z3Xi7Jm7cHtuQga9L9DUfGVQhLB2apL3CHtkwPPf1io=; b=AxJrxxPQCGuwZjag4WbAzjmhYxRaPycShKkLyk2SEJh/RHxau5t7RYcEKFVboBjTU9354WUfpw9CiZbr74nBIryg5DCD/A68CoAUdVWf6vQN85aQJXnAiRfrNhm0TNzDMjoQHI7b6AKai5Z4UksJJAdXvpuX5CjNTyjjHHkIF/ZCL7MbgURDhqxGuu1BfAs15YGyQIwyVhdTWlibDAqOBdic9OcQKd2H8fYoZD8TqTy+1AmGx7G6AZpf8W5g1hlw/nSyw92zENWw/YmU603U3lTGRVX42Qj1guqeRGn8C1ILhLLbhYP3BAtpSP/91xbribnEx7cwNXDtCsy/oRwRSQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.35) smtp.rcpttodomain=marvell.com smtp.mailfrom=nvidia.com; dmarc=pass (p=quarantine sp=none pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=z3Xi7Jm7cHtuQga9L9DUfGVQhLB2apL3CHtkwPPf1io=; b=QG0sg1VAysxZWuqopAVFw6PJpaipDqQ9hXhqHc2CpMhhAvP6ZCF+06Xl4nqlPaGhM96jn5U95s5aGSK2LY2f9m2E8ple6R0UPHupKmjSweHX2qwWQIS5bfbKruwgrDgJyy4Xra28BuFHn6vN90ux19ryUNO056a99mHQ9YheyqW5pRlVhv4mwYIowPIwXRMSnDqBR9e735yQlFESkirri/ISICbUN6N/KdfGpHGzw0ZmgWWcjQIyC7n+WPMAt0Cq2aYL3kRz/ObJMYBfwzjK1T7f6piVBs1H7R6ubr+4LY2fcSd7ru7Re6KNUvlx4drTy6TZNsk8ZDMvclt8yYl5+Q== Received: from DM5PR10CA0011.namprd10.prod.outlook.com (2603:10b6:4:2::21) by CH2PR12MB4183.namprd12.prod.outlook.com (2603:10b6:610:7a::24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4331.24; Thu, 22 Jul 2021 11:06:30 +0000 Received: from DM6NAM11FT058.eop-nam11.prod.protection.outlook.com (2603:10b6:4:2:cafe::29) by DM5PR10CA0011.outlook.office365.com (2603:10b6:4:2::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4352.24 via Frontend Transport; Thu, 22 Jul 2021 11:06:30 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.35) smtp.mailfrom=nvidia.com; marvell.com; dkim=none (message not signed) header.d=none; marvell.com; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.35 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.35; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.35) by DM6NAM11FT058.mail.protection.outlook.com (10.13.172.216) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4352.24 via Frontend Transport; Thu, 22 Jul 2021 11:06:30 +0000 Received: from HQMAIL107.nvidia.com (172.20.187.13) by HQMAIL111.nvidia.com (172.20.187.18) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Thu, 22 Jul 2021 11:06:29 +0000 Received: from vdi.nvidia.com (172.20.187.5) by mail.nvidia.com (172.20.187.13) with Microsoft SMTP Server id 15.0.1497.2 via Frontend Transport; Thu, 22 Jul 2021 11:06:24 +0000 From: Boris Pismenny To: , , , , , , , , , , CC: , , , , , Subject: [PATCH v5 net-next 28/36] mlx5: Add sq state test bit for nvmeotcp Date: Thu, 22 Jul 2021 14:03:17 +0300 Message-ID: <20210722110325.371-29-borisp@nvidia.com> X-Mailer: git-send-email 2.24.1 In-Reply-To: <20210722110325.371-1-borisp@nvidia.com> References: <20210722110325.371-1-borisp@nvidia.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 3e23eaa1-fd0e-4fe9-7b9e-08d94d00c236 X-MS-TrafficTypeDiagnostic: CH2PR12MB4183: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:758; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: r75aQZjRqbawv/5XgB31v2O4zrzVpSNLwP6jHXhTRkWK3Bc/TCt7jt+kpOFUkyWRC7efFd8U1Ous9fxCBSA5NqiulXDf6X6fASg4HcmI68h3V/Sg0M9GAzyMA5LuBCtaaUfuUjJ8UPdd8FosQtt6Qt+4eLulH2EF9yyZqi6QWe4tRlRxVsvvKRZXVFwF6lw46ZZhhG0M2uG8O5eU3eqyTkFwBWH79Hzhv6OdRUk7P61JQg9q5ZlWw8UdZvdt363zvOpPY95C9XjbSEUsuRcLJNGgi36Gk3iLNCQ/ctKnaVUkZkZsvdUOPSkkr1SczZM7rLjUxfKuIKNkdmokzKcKj9KtcZow2+RikLJb6KjFN4W+npVsZbTvpxjD2QkUxwwoqbwF833yG7+TB+J6orDXzGybHI0EDgKo+vLL7USW3N3qtdEp/0g7QTzAj6fCDg4wMwsGqxUiOfasTs/IuMwpGTH+4ODEPdbo7vFo7dAGZ5jgXBrV5bTWs7uIgXkIiSkcGpqwuaF2h3V96k/gQWzHjdWkC5Nnu4pkG6JxBCSw8XWlSiOoG0zFnIIe8+ZYYEsFbqm2ke1I6X5vHxTLZsZNQmW9oMrZw51WRN3DqM9WPV6VRLqETRUGMzsvuyAzG2ZgGZFpR3YhCUBxqOrq/QQ8CzW4kkZBq8W8FhnDLyBi9B6VDUGrFtoEEZJ/YjRlRHi/y1/01mOUwtiqr8z/rS9tkp5O/lj7GfZYTZen/UQIW5k= X-Forefront-Antispam-Report: CIP:216.228.112.35; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:schybrid02.nvidia.com; CAT:NONE; SFS:(4636009)(46966006)(36840700001)(5660300002)(356005)(70586007)(36756003)(1076003)(107886003)(26005)(86362001)(36906005)(70206006)(6666004)(316002)(4326008)(7636003)(8676002)(82310400003)(54906003)(36860700001)(8936002)(110136005)(2616005)(47076005)(7696005)(336012)(186003)(921005)(2906002)(7416002)(508600001)(426003); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Jul 2021 11:06:30.2205 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 3e23eaa1-fd0e-4fe9-7b9e-08d94d00c236 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.112.35]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DM6NAM11FT058.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH2PR12MB4183 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Yoray Zack Add to the sq state a bit indicate if their is an nvmeotcp crc tx offload. Signed-off-by: Yoray Zack --- drivers/net/ethernet/mellanox/mlx5/core/en.h | 1 + drivers/net/ethernet/mellanox/mlx5/core/en_main.c | 2 ++ 2 files changed, 3 insertions(+) diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en.h b/drivers/net/ethernet/mellanox/mlx5/core/en.h index c87f32492ea4..58724d4c27e1 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en.h +++ b/drivers/net/ethernet/mellanox/mlx5/core/en.h @@ -353,6 +353,7 @@ enum { MLX5E_SQ_STATE_VLAN_NEED_L2_INLINE, MLX5E_SQ_STATE_PENDING_XSK_TX, MLX5E_SQ_STATE_PENDING_TLS_RX_RESYNC, + MLX5E_SQ_STATE_NVMEOTCP, }; struct mlx5e_tx_mpwqe { diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c index ecb12c7fdb7d..b76e590c237b 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c @@ -1142,6 +1142,8 @@ static int mlx5e_alloc_txqsq(struct mlx5e_channel *c, set_bit(MLX5E_SQ_STATE_VLAN_NEED_L2_INLINE, &sq->state); if (MLX5_IPSEC_DEV(c->priv->mdev)) set_bit(MLX5E_SQ_STATE_IPSEC, &sq->state); + if (MLX5_CAP_DEV_NVMEOTCP(c->priv->mdev, crc_tx)) + set_bit(MLX5E_SQ_STATE_NVMEOTCP, &sq->state); if (param->is_mpw) set_bit(MLX5E_SQ_STATE_MPWQE, &sq->state); sq->stop_room = param->stop_room; From patchwork Thu Jul 22 11:03:18 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Boris Pismenny X-Patchwork-Id: 485098 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-20.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER, INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C5350C63793 for ; Thu, 22 Jul 2021 11:06:42 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id A39316120D for ; Thu, 22 Jul 2021 11:06:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231771AbhGVK0G (ORCPT ); Thu, 22 Jul 2021 06:26:06 -0400 Received: from mail-dm6nam12on2062.outbound.protection.outlook.com ([40.107.243.62]:40131 "EHLO NAM12-DM6-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S231830AbhGVK0C (ORCPT ); Thu, 22 Jul 2021 06:26:02 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=NPQYS7auD4l8+XRDQ/7tNcqsm61hffHbYOF1YZEUeNKOlnv9ZQZkaXjQ4/VYxy3q/MeKl+QkqMTI+E6zm5U6ONiTRfOQ7hN9DCqNHTB55LuJRH6qYB7rUEVyvm3+HRUG+rlVKLRpkmi/y1culVP9RyE6/FkIceBkAUb7xAih7YrRjMLq8MvkMEj3r6HKG5LaSfK4DxyI1B5M9NgQAJ/5+aEJIz2+e0zgiD8MNlreCdgUroz91Z1cmsyOUxoR2vErqgSMN8NLscDgi+YBAXKdILWU3Mw7zPiDP8r0JsbQyBejX4K4WJ3JLyEfUZwDYCw4wxCEq5KQCZFLtdVQSfZTkw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=vFWAURlZn+oTTIbQ4EusZH6LHrHcGZ+656qWJEmoB20=; b=Qz4wBSK37w8aLr802+RaXZL5A0vKY2ZMjYv2PzUhJjpCN4jQ1d72qb58h+0O5PEGVJ6ygAkFdcU1lUnIb/GYgOX8CfJJi5/CIvHJjtmhygSodQrRJ9gmFMHxdcvVf0imPHIRHFb8UK0ro5vDNbLmd0y1yBnAmrq9QLV+T5WuUOsYc5mVQUTV0w6FGz9ky2Vj3D4XS3HUBjzGtm6ghsHv1mBG1BzpWIh/eRbr+sKz/Ttgy6MghNDACFX4VRRVwUvGVoUZPVQPT+6ZPo8rsXWDRA5MZN/+sGBSvWbG6Y0HluJmKKZyqikSj6TimdqQFrJDQAU2PmZ217bLt+lwLYhSCA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.34) smtp.rcpttodomain=marvell.com smtp.mailfrom=nvidia.com; dmarc=pass (p=quarantine sp=none pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=vFWAURlZn+oTTIbQ4EusZH6LHrHcGZ+656qWJEmoB20=; b=Kevom7opTg5whHCR4fmxlSzg5IhtPHzybrIFnMqwvOB5QahZP+I9B3cEMsP+kunnWB1pUTbDgIMDU0MFweyPBAFf2UUdxcRcT8Z6MAWNMbbDM+Cz6thOv4tprZf1UMNG5kOrcwmJJTrUE2tp5in1JnHATkHqJscbvNL+wR/TcuTCj9yhUTFibiH7s3i5PayhrERaNSyogG9RVhQzw7tZVvHE+b3/1cQybv3SrngZsV2/8YIChxg7hrLVc5UwNE3MrFOEgTM8gFdhjkecoZe14ELv8ChLLnxC9d9lT7qKuocqCRkmPSBj4WiVlaRE/VYbKnFByQQPxwcD0XdANVpxvg== Received: from BN8PR12CA0021.namprd12.prod.outlook.com (2603:10b6:408:60::34) by DM6PR12MB3035.namprd12.prod.outlook.com (2603:10b6:5:3a::24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4331.23; Thu, 22 Jul 2021 11:06:36 +0000 Received: from BN8NAM11FT062.eop-nam11.prod.protection.outlook.com (2603:10b6:408:60:cafe::91) by BN8PR12CA0021.outlook.office365.com (2603:10b6:408:60::34) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4352.25 via Frontend Transport; Thu, 22 Jul 2021 11:06:35 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.34) smtp.mailfrom=nvidia.com; marvell.com; dkim=none (message not signed) header.d=none; marvell.com; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.34 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.34; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.34) by BN8NAM11FT062.mail.protection.outlook.com (10.13.177.34) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4352.24 via Frontend Transport; Thu, 22 Jul 2021 11:06:35 +0000 Received: from HQMAIL107.nvidia.com (172.20.187.13) by HQMAIL107.nvidia.com (172.20.187.13) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Thu, 22 Jul 2021 11:06:35 +0000 Received: from vdi.nvidia.com (172.20.187.5) by mail.nvidia.com (172.20.187.13) with Microsoft SMTP Server id 15.0.1497.2 via Frontend Transport; Thu, 22 Jul 2021 11:06:30 +0000 From: Boris Pismenny To: , , , , , , , , , , CC: , , , , , Subject: [PATCH v5 net-next 29/36] mlx5: Add support to NETIF_F_HW_TCP_DDP_CRC_TX feature Date: Thu, 22 Jul 2021 14:03:18 +0300 Message-ID: <20210722110325.371-30-borisp@nvidia.com> X-Mailer: git-send-email 2.24.1 In-Reply-To: <20210722110325.371-1-borisp@nvidia.com> References: <20210722110325.371-1-borisp@nvidia.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: aa6f2228-9ac5-4409-2b78-08d94d00c577 X-MS-TrafficTypeDiagnostic: DM6PR12MB3035: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:346; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: mTh8PngNsRmpAoKQQJRU6cZ4xS+jahvvj358diA27WmmZAuub+ebFYxvMv06Ls1M+u9PEbOfD2kOUz4HqMVLCCdB0OGkmyhWJrH8diOt+qK5urftl4KpMq61x1r5afI0x9GuvaxVv0UzVgMq2Zd3CMp46S/L2o+xdW3vBoz82OnwA/+qKrXM2kPlU8XYwLz8Ua0Th1O1hloa1srATeLDg/gA6D4IQMh2JiuutX8hvRgRkz15pGc+shYqVe5y6WPcOXGhC72oUZTh1ZhN21VvmoL2hSe1VxheXuFETNa7JFwDREIsjNQ7i5w3FZI72znL11J5aA1Dym42BEGhiHAMFGpGo1wDEFf6BbzshXbg9FMvMioBHN9lrM4WhhLKKpExpeSJcz6H8CSV3npk2BGASGbCd6l9CIN6UsAg+Ps2yfY2MI7CBLMJ7pj/e38V1S1TJjxeqS6VZeGChRYlV6boU7K2ifh+R4/0ASuopwF3h42pAbOfxxrBTkqVhF3pn5+PcUZgsoQH9ElUu6m0pB4XuV+g3nh3WFGpLozPMd+hv618pWqgsvsy7KbU5lFh7u8sDlKcsWbNGo1sEQLwL4YXhj4/Te2AMenx2N2aPCi8nC1G0AiOz7P4N+1UQh8YT8rUcsNXfbxqKV8Fv1pJ9nY6X7luUCj+K2aRdlFBL1coTbQIM0tzXr3fiY0qbtqdWFux9P+M73fzqPkr8uJhnDoIOQ== X-Forefront-Antispam-Report: CIP:216.228.112.34; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:schybrid03.nvidia.com; CAT:NONE; SFS:(4636009)(396003)(376002)(136003)(346002)(39860400002)(46966006)(36840700001)(5660300002)(8936002)(70586007)(1076003)(356005)(36756003)(4326008)(107886003)(7696005)(26005)(316002)(36906005)(6666004)(70206006)(86362001)(7636003)(8676002)(82310400003)(54906003)(110136005)(2616005)(186003)(36860700001)(921005)(478600001)(4744005)(82740400003)(2906002)(426003)(47076005)(7416002)(336012); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Jul 2021 11:06:35.6227 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: aa6f2228-9ac5-4409-2b78-08d94d00c577 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.112.34]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BN8NAM11FT062.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR12MB3035 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Yoray Zack Signed-off-by: Yoray Zack --- drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c index 7f6607cac9fb..db6ca734d129 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c @@ -961,6 +961,11 @@ void mlx5e_nvmeotcp_build_netdev(struct mlx5e_priv *priv) netdev->hw_features |= NETIF_F_HW_ULP_DDP; } + if (MLX5_CAP_DEV_NVMEOTCP(priv->mdev, crc_tx)) { + netdev->features |= NETIF_F_HW_ULP_DDP; + netdev->hw_features |= NETIF_F_HW_ULP_DDP; + } + netdev->ulp_ddp_ops = &mlx5e_nvmeotcp_ops; } From patchwork Thu Jul 22 11:03:19 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Boris Pismenny X-Patchwork-Id: 484311 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-20.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER, INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 581D1C6377D for ; Thu, 22 Jul 2021 11:06:49 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 3A53D60725 for ; Thu, 22 Jul 2021 11:06:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231817AbhGVK0N (ORCPT ); Thu, 22 Jul 2021 06:26:13 -0400 Received: from mail-co1nam11on2046.outbound.protection.outlook.com ([40.107.220.46]:41185 "EHLO NAM11-CO1-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S231693AbhGVK0I (ORCPT ); Thu, 22 Jul 2021 06:26:08 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Pb97q/s1+ZXHF0lcWUHy8CJe1Qxst49HeMkznwLrTkRJhPhdvy/y40WeMJE+zRACsvEH1qiNW7zR7RQ0Wv10esQRPM72U9udQdUkMncQRt/mn1Qt8t6Ze62XG5FAGgz54y1ELVja0S+tSOauMHLCb3eiuLvT1lkWvj4q8mzNF/Di7KDnZxlR+wnf+naSz4i9SweJiKiUO07tUHATh7x12hwpRYrSK4yeh5jUXy698g5B921UanwxYbqtTeENUz6C5Mrm+Hi6RNqfA55aZ6qMINiyqH0Y8bNuqcYNxH2cZ27Al4VDRV/Viru2xXsN5Ki8vnhTITLIMCDWoi/pnAyfqA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=6hgX2eAKk6eToFJ+7VZTHz3wQjJMmNFuvM8vG0/psnk=; b=C1C3qp2xD8lUC8vdvI9r7+d2rcoW6ZK5sg+J4MeT1PU+rEyjrH3jy23cXWrtCDoEKJFXPOXra/zxty2aXl2lJWbPIrDpIlEOmhGZNOxUtNguGYlW+WTRWB4c1904R0wIV5KMjpod1+wxRm3zxP/9bm8PSEFA9QIzxbjnOLifX4d8TM//EPunDML8sIaY/cy0iGIEpz0iyexplRAFJccW2XyXhTFPo2yTJ3A7EKHjPYbN6+QuJAyUveSpawVSCzeTLkZcrym25jc4kwcSN2rVXuGV60w5OAIkagIM39zBTCAG/AP24S1KzyAO4dCpbtZqNqCDUBFfGn3m2ewRYjiRyw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.32) smtp.rcpttodomain=marvell.com smtp.mailfrom=nvidia.com; dmarc=pass (p=quarantine sp=none pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=6hgX2eAKk6eToFJ+7VZTHz3wQjJMmNFuvM8vG0/psnk=; b=UgxAd9wiuIA4Dt/7+qcj2+SYtjUEMFeG0swVCAUmbShZd9SqxkALDx/I9E5Xtpq+10JCg2Biz1pAAFNR83ihOSDb0rvW9w9+K/ZyLrx9ULkxi/0j8PAPPj2d22bbCFvA138I7LDSIPVjJYbffRCS63ecnmOLVg8masQVb6nU6QCtZ8w1Ziqltyqd3r23LKhw8VJQEWBn3wA4dEW+IA9/cb5/s9EfQQRsFZjVBhrmiSD4eLJLhEGmZnen/dIliGpMb5UXdl+TLoSubP80DQjBPE3Wn69YbU4XGTmEt9N3PXDsBQQvFxj2JK7b+0KmQVIyeWcVKHNqzfLk282imthA7w== Received: from DM3PR12CA0138.namprd12.prod.outlook.com (2603:10b6:0:51::34) by MN2PR12MB4519.namprd12.prod.outlook.com (2603:10b6:208:262::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4331.24; Thu, 22 Jul 2021 11:06:42 +0000 Received: from DM6NAM11FT020.eop-nam11.prod.protection.outlook.com (2603:10b6:0:51:cafe::83) by DM3PR12CA0138.outlook.office365.com (2603:10b6:0:51::34) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4352.26 via Frontend Transport; Thu, 22 Jul 2021 11:06:42 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.32) smtp.mailfrom=nvidia.com; marvell.com; dkim=none (message not signed) header.d=none; marvell.com; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.32 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.32; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.32) by DM6NAM11FT020.mail.protection.outlook.com (10.13.172.224) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4352.24 via Frontend Transport; Thu, 22 Jul 2021 11:06:41 +0000 Received: from HQMAIL107.nvidia.com (172.20.187.13) by HQMAIL109.nvidia.com (172.20.187.15) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Thu, 22 Jul 2021 04:06:40 -0700 Received: from vdi.nvidia.com (172.20.187.5) by mail.nvidia.com (172.20.187.13) with Microsoft SMTP Server id 15.0.1497.2 via Frontend Transport; Thu, 22 Jul 2021 11:06:35 +0000 From: Boris Pismenny To: , , , , , , , , , , CC: , , , , , Subject: [PATCH v5 net-next 30/36] net/mlx5e: NVMEoTCP DDGST TX offload TIS Date: Thu, 22 Jul 2021 14:03:19 +0300 Message-ID: <20210722110325.371-31-borisp@nvidia.com> X-Mailer: git-send-email 2.24.1 In-Reply-To: <20210722110325.371-1-borisp@nvidia.com> References: <20210722110325.371-1-borisp@nvidia.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 4e535892-c5af-4d47-cccb-08d94d00c917 X-MS-TrafficTypeDiagnostic: MN2PR12MB4519: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:25; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: fRhrv8gPZ6lYjbeXOtH6dg0sZFzA9KTCh9NEWJXhjS6Dia3ERscy6eAONkH5vV4w8F0bF/grEJN+/x/Qc0tW57qxrzc4G8HlMoOzb1wX4KlgDmy93hLp3+MaLHpZvNmIdPyznqDZsfjzdUeMRzl8CmP0UI7uMRmxXmq14E4hnihdnpvVIl38ZyF+ROWgNKmUtdQFxddHdpR9FKn1l0TitMN+NpXx6slZvoCq7st5xw80Ax6c4+ZYkji6DNM/YDoDuJjejtJEgpM19OmPVxFnPxEZ1WPS5xk3jUrnLwD6Gr0QeGGKmw8O3UkXDWFY6eTG+SBlsIiMnE01QqPaqDTwIW2gXXKtfNwq62XFNgBhsINpJFWL68iasYYvXZxUxQGsyl8D+0vSQQAI77e8jQE8qVbUtYruaCwCwaAnlYavH9t28SWl1rMoMZLcS3F6OXDf3Ve4Kud6hliFg1Y8XEvVku8HniV4KUt9ud1iOY0lgO19HGXNNoW5hQ7p3iaGF+LnhRt+b9FCcj1kOcHQEMj37SZnOJFEffkVyPEYthFHEPfOegE8BzCB+E2RQc30XxymmU8QfQmVK6bYdicGZP0nxYz4CmTwIURwXR9kFaqE9xtN6lXUlwTfBpZZSPrlSfJR/jl6cLcqhDn7QU1K1TcvQoOtOh0lg7QwOzXyCU0VnL3aHQRCgB1iMD9b6cOENR/ROaJEdJkYTMfEK1FnkUCacJpDketeprA2tV3j7uPlNyo= X-Forefront-Antispam-Report: CIP:216.228.112.32; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:schybrid01.nvidia.com; CAT:NONE; SFS:(4636009)(39860400002)(396003)(376002)(346002)(136003)(46966006)(36840700001)(2616005)(426003)(6666004)(70586007)(110136005)(186003)(5660300002)(356005)(83380400001)(478600001)(4326008)(8676002)(7696005)(8936002)(36860700001)(82310400003)(316002)(107886003)(336012)(36756003)(54906003)(70206006)(82740400003)(2906002)(7416002)(921005)(1076003)(86362001)(26005)(7636003)(47076005); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Jul 2021 11:06:41.7655 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 4e535892-c5af-4d47-cccb-08d94d00c917 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.112.32]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DM6NAM11FT020.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR12MB4519 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Yoray Zack NVMEoTCP DDGST Tx offload needs TIS. This commit add the infrastructer for this TIS. Signed-off-by: Yoray Zack --- .../mellanox/mlx5/core/en_accel/nvmeotcp.c | 17 +++++++++++++++++ .../net/ethernet/mellanox/mlx5/core/en_main.c | 2 +- include/linux/mlx5/mlx5_ifc.h | 3 ++- 3 files changed, 20 insertions(+), 2 deletions(-) diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c index db6ca734d129..d42f346ac8f5 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c @@ -35,6 +35,11 @@ static void mlx5e_nvmeotcp_destroy_tir(struct mlx5e_priv *priv, int tirn) mlx5_core_destroy_tir(priv->mdev, tirn); } +static void mlx5e_nvmeotcp_delete_tis(struct mlx5e_priv *priv, int tisn) +{ + mlx5_core_destroy_tis(priv->mdev, tisn); +} + static inline u32 mlx5e_get_channel_ix_from_io_cpu(struct mlx5e_priv *priv, u32 io_cpu) { @@ -137,6 +142,18 @@ void mlx5_destroy_nvmeotcp_tag_buf_table(struct mlx5_core_dev *mdev, u32 uid) mlx5_cmd_exec(mdev, in, sizeof(in), out, sizeof(out)); } +static int mlx5e_nvmeotcp_create_tis(struct mlx5_core_dev *mdev, u32 *tisn) +{ + u32 in[MLX5_ST_SZ_DW(create_tis_in)] = {}; + void *tisc; + + tisc = MLX5_ADDR_OF(create_tis_in, in, ctx); + + MLX5_SET(tisc, tisc, nvmeotcp_en, 1); + + return mlx5e_create_tis(mdev, in, tisn); +} + #define MLX5_CTRL_SEGMENT_OPC_MOD_UMR_TIR_PARAMS 0x2 #define MLX5_CTRL_SEGMENT_OPC_MOD_UMR_NVMEOTCP_TIR_STATIC_PARAMS 0x2 #define MLX5_CTRL_SEGMENT_OPC_MOD_UMR_UMR 0x0 diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c index b76e590c237b..2a9718d3c2d3 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c @@ -3152,7 +3152,7 @@ int mlx5e_create_tis(struct mlx5_core_dev *mdev, void *in, u32 *tisn) MLX5_SET(tisc, tisc, transport_domain, mdev->mlx5e_res.hw_objs.td.tdn); - if (MLX5_GET(tisc, tisc, tls_en)) + if (MLX5_GET(tisc, tisc, tls_en) || MLX5_GET(tisc, tisc, nvmeotcp_en)) MLX5_SET(tisc, tisc, pd, mdev->mlx5e_res.hw_objs.pdn); if (mlx5_lag_is_lacp_owner(mdev)) diff --git a/include/linux/mlx5/mlx5_ifc.h b/include/linux/mlx5/mlx5_ifc.h index f0310c24f408..a4965bf1e607 100644 --- a/include/linux/mlx5/mlx5_ifc.h +++ b/include/linux/mlx5/mlx5_ifc.h @@ -3300,7 +3300,8 @@ struct mlx5_ifc_traffic_counter_bits { struct mlx5_ifc_tisc_bits { u8 strict_lag_tx_port_affinity[0x1]; u8 tls_en[0x1]; - u8 reserved_at_2[0x2]; + u8 nvmeotcp_en[0x1]; + u8 reserved_at_3[0x1]; u8 lag_tx_port_affinity[0x04]; u8 reserved_at_8[0x4]; From patchwork Thu Jul 22 11:03:20 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Boris Pismenny X-Patchwork-Id: 485097 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-20.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER, INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 42DE3C63793 for ; Thu, 22 Jul 2021 11:06:54 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 2AF6C6121E for ; Thu, 22 Jul 2021 11:06:54 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231675AbhGVK0R (ORCPT ); Thu, 22 Jul 2021 06:26:17 -0400 Received: from mail-bn1nam07on2055.outbound.protection.outlook.com ([40.107.212.55]:35134 "EHLO NAM02-BN1-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S231670AbhGVK0O (ORCPT ); Thu, 22 Jul 2021 06:26:14 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=MaLb2VtSoBv5PfQrXQxn9DyfQXDjLfzx4g5SZ7gnsl/Fa5LjDl/IaRW70xwiZYtavTGknJfWlGKc9OiftrOkbgH68nNnztSXDnD5KO+uw8c9XftRKfmDWgv35gmCUbEXAOwZ7G85TyBxj8ReJeJdI9VMzOJ0z6CcEoUjw2n7LpnHjM+7b/Sl8AkQ7CDjSr7RDxz4VONX52XBornYxU9pSP7Djw6GybpC0Iqa0O0/321CkP25SzZ8Ql5F9TiA3F2jD4b7+o6V6IFzazyXUipl4fUDBxj6xrWIseTpQQ5IV/dZmT17angg0d6B3aHMGBITgxYZFzyT0jwJbR1AZMA1QQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=/Vsen5bhy2iMXAw5gSfBDTtXKWFC9pBrUrotXQM4q40=; b=Vb9xXCJs8uOU4KCkUCWKhipFqKm2rhjaxFbjXBV8oCTD5T4XzzDo8abXnqApvsyWrPsayQIeiX+GlYPWCjHVciOZriWET58DBjqY/5RS8m4+KqMIHif15a5q6s5hVNt2YQUAi2TEvSlv3YPk0yXqNrOLtmmm0zDSqcN3bd+WAJzUgkFzhLKyfoYhzCabJyU33PkwBaC5EkHqZQDdsMHXAi6HvKZRGErEpdUAi5kXU783NlxXHfQ9j81HJXDhJ6OZupBAr8vo1TriiTFQkITgKZztNls8YB+6Guc0lVXT6Mqd16ZS4XVQE8PR0BfnP7sXPfF7q0BK20LxrsR+ahKT+w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.35) smtp.rcpttodomain=marvell.com smtp.mailfrom=nvidia.com; dmarc=pass (p=quarantine sp=none pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=/Vsen5bhy2iMXAw5gSfBDTtXKWFC9pBrUrotXQM4q40=; b=ahv8U6rpesu0L9yQELROfWNnxx7gN/YTLZtxNuMAEJLURO7+KfuukGy7kub95C+Ge7FJJNkBTjMW6e3Zs0Qya90IRKm26JHzsmfZIjwTWCc+owgEap7yfRTaHDpv/ciWT7XMiiugql2yBPb9CKAtCWvqg3+3tDNZEkIrA/oqkTx+gEBK4ijKgPU33Hhpp1UuBttV3slEaw+YIC6nYh0f21p0kH3aXU7dCXJmgCYuTGGQx5dnwKSZTSQWhoZL1Sf/ROiArTnwLfePs/DA59h3HjkUl+U7TZw2v7N5qgLntLqDnOu1tdfWRQwfpd1EM+X9TAciGjPGvNFxV51sztehsQ== Received: from DM5PR12CA0068.namprd12.prod.outlook.com (2603:10b6:3:103::30) by BYAPR12MB2743.namprd12.prod.outlook.com (2603:10b6:a03:61::28) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4331.23; Thu, 22 Jul 2021 11:06:48 +0000 Received: from DM6NAM11FT009.eop-nam11.prod.protection.outlook.com (2603:10b6:3:103:cafe::84) by DM5PR12CA0068.outlook.office365.com (2603:10b6:3:103::30) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4352.26 via Frontend Transport; Thu, 22 Jul 2021 11:06:48 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.35) smtp.mailfrom=nvidia.com; marvell.com; dkim=none (message not signed) header.d=none; marvell.com; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.35 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.35; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.35) by DM6NAM11FT009.mail.protection.outlook.com (10.13.173.20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4352.24 via Frontend Transport; Thu, 22 Jul 2021 11:06:47 +0000 Received: from HQMAIL107.nvidia.com (172.20.187.13) by HQMAIL111.nvidia.com (172.20.187.18) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Thu, 22 Jul 2021 11:06:46 +0000 Received: from vdi.nvidia.com (172.20.187.5) by mail.nvidia.com (172.20.187.13) with Microsoft SMTP Server id 15.0.1497.2 via Frontend Transport; Thu, 22 Jul 2021 11:06:41 +0000 From: Boris Pismenny To: , , , , , , , , , , CC: , , , , , Subject: [PATCH v5 net-next 31/36] net/mlx5e: NVMEoTCP DDGST Tx offload queue init/teardown Date: Thu, 22 Jul 2021 14:03:20 +0300 Message-ID: <20210722110325.371-32-borisp@nvidia.com> X-Mailer: git-send-email 2.24.1 In-Reply-To: <20210722110325.371-1-borisp@nvidia.com> References: <20210722110325.371-1-borisp@nvidia.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: e33b59a9-fa94-4fee-141e-08d94d00ccc9 X-MS-TrafficTypeDiagnostic: BYAPR12MB2743: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:164; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: xxAVwlP6OO+eSmTy8dmDdUXuJtqK0QR7Mv2Q15HgjZn2to/nrruJxHLDq+OViXw/KszeXw2yQhgwF0dmJTW+eKbiqYXeGLtZAzgcOjTw0878ngeZlToMdRqjjuMHKE8qTXnizsvfz/WrbRyop0Tpiu9pc/GGLqNk+Eo2uQn+cgZsGkoUrjgujoeZRWWtQ/L63E392F0dhlcpMeasNUbTaqOpauTmH//VbVPESWnmH0dS4MLsc3S0AzD/ZHUf7fLQeOcW6vyY+wD/3EPqwuLyOSF3j6BKJH1XFrQfXPtbMiULrGdXL3LeBsnVBbYcR4HM+XDivHbLht3UFicUU1p1v6m7I6rx2xlCE/LbrArzQ4eOTJ4pZ5w77NnENZGNTvzzR8/+fLIPev0UcXJd+XqOwAAxGVmWycUcg8M969kVdPiCI/BazX4/CXq7wVI6rg5kT70JZjTOqqwSDLCvxISDDslhWpdVHKuKvJXP/bVByxG7m1F0vgGBean6jDpDd6+BM+pf5PD7lhKL/HkDZeoJ5PdWDHozWWpfeUYqPBhiVOr4iwQ0N+Q75BdXr9cbBPnWiDaxL4pxCMvdHsy7y9R95kny7clhpatOzqU+GNQjUXj3TAeEtiOrEekxm5hAl9dHLwfTullGrAeT2oUYFTApZDSZ5E0GZzasrQDgx1q7MlnCBUq6eVEUrteQNc4dRFxcadDuJdxEWfARGTE+YLgITITy3gLREd0uSSq1/zOoqJA= X-Forefront-Antispam-Report: CIP:216.228.112.35; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:schybrid04.nvidia.com; CAT:NONE; SFS:(4636009)(396003)(376002)(346002)(136003)(39860400002)(36840700001)(46966006)(110136005)(8936002)(316002)(426003)(336012)(70206006)(4326008)(8676002)(36906005)(2906002)(1076003)(186003)(7696005)(70586007)(107886003)(36756003)(7416002)(26005)(7636003)(2616005)(54906003)(82740400003)(6666004)(82310400003)(921005)(47076005)(356005)(5660300002)(478600001)(36860700001)(86362001)(83380400001); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Jul 2021 11:06:47.9345 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: e33b59a9-fa94-4fee-141e-08d94d00ccc9 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.112.35]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DM6NAM11FT009.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR12MB2743 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Yoray Zack This commit add support for DDGST TX offload to the mlx5e_nvmeotcp_queue_init/teardown function. If enable, mlx5e_nvmeotcp_queue_init will call mlx5e_nvmeotcp_queue_tx_init to handle TX offload init. Add to mlx5e NVMEoTCP queue is responsible for: - Create a separate TIS to identify the queue and maintain the HW context - Update ulp_ddp_ctx params. Signed-off-by: Yoray Zack --- .../mellanox/mlx5/core/en_accel/nvmeotcp.c | 47 +++++++++++++++++++ .../mellanox/mlx5/core/en_accel/nvmeotcp.h | 12 ++++- 2 files changed, 58 insertions(+), 1 deletion(-) diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c index d42f346ac8f5..6023e1ae7be4 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c @@ -669,6 +669,36 @@ mlx5e_nvmeotcp_queue_rx_init(struct mlx5e_nvmeotcp_queue *queue, return err; } +static int +mlx5e_nvmeotcp_queue_tx_init(struct mlx5e_nvmeotcp_queue *queue, + struct mlx5_core_dev *mdev, + struct net_device *netdev) +{ + struct sock *sk = queue->sk; + int err, tisn; + + err = mlx5e_nvmeotcp_create_tis(mdev, &tisn); + + if (err) { + mlx5_core_err(mdev, "create tis failed, %d\n", err); + return err; + } + + queue->tisn = tisn; + queue->ulp_ddp_ctx.expected_seq = tcp_sk(sk)->write_seq; + queue->pending = true; + queue->end_seq_hint = 0; + queue->ulp_ddp_ctx.netdev = netdev; + queue->ulp_ddp_ctx.ddgst_len = 4; + + /* following this assignment mlx5e_nvmeotcp_is_sk_tx_device_offloaded + * will return true and ulp_ddp_ctx might be accessed + * by the netdev's xmit function. + */ + smp_store_release(&sk->sk_validate_xmit_skb, ulp_ddp_validate_xmit_skb); + return err; +} + #define OCTWORD_SHIFT 4 #define MAX_DS_VALUE 63 static int @@ -680,6 +710,8 @@ mlx5e_nvmeotcp_queue_init(struct net_device *netdev, bool crc_rx = ((netdev->features & NETIF_F_HW_ULP_DDP) && (config->dgst & NVME_TCP_DATA_DIGEST_ENABLE)); bool zerocopy = (netdev->features & NETIF_F_HW_ULP_DDP); + bool crc_tx = (config->dgst & NVME_TCP_DATA_DIGEST_ENABLE) && + (netdev->features & NETIF_F_HW_ULP_DDP); struct mlx5e_priv *priv = netdev_priv(netdev); struct mlx5_core_dev *mdev = priv->mdev; struct mlx5e_nvmeotcp_queue *queue; @@ -709,6 +741,7 @@ mlx5e_nvmeotcp_queue_init(struct net_device *netdev, goto free_queue; } + queue->crc_tx = crc_tx; queue->crc_rx = crc_rx; queue->zerocopy = zerocopy; queue->ulp_ddp_ctx.type = ULP_DDP_NVME; @@ -736,6 +769,12 @@ mlx5e_nvmeotcp_queue_init(struct net_device *netdev, if (err) goto destroy_rx; + if (crc_tx) { + err = mlx5e_nvmeotcp_queue_tx_init(queue, mdev, netdev); + if (err) + goto remove_queue_from_hash; + } + stats->nvmeotcp_queue_init++; write_lock_bh(&sk->sk_callback_lock); ulp_ddp_set_ctx(sk, queue); @@ -743,6 +782,9 @@ mlx5e_nvmeotcp_queue_init(struct net_device *netdev, refcount_set(&queue->ref_count, 1); return err; +remove_queue_from_hash: + rhashtable_remove_fast(&priv->nvmeotcp->queue_hash, + &queue->hash, rhash_queues); destroy_rx: if (zerocopy || crc_rx) mlx5e_nvmeotcp_destroy_rx(queue, mdev, zerocopy); @@ -778,6 +820,11 @@ mlx5e_nvmeotcp_queue_teardown(struct net_device *netdev, rhashtable_remove_fast(&priv->nvmeotcp->queue_hash, &queue->hash, rhash_queues); ida_simple_remove(&priv->nvmeotcp->queue_ids, queue->id); + if (queue->crc_tx) { + smp_store_release(&sk->sk_validate_xmit_skb, NULL); + mlx5e_nvmeotcp_delete_tis(priv, queue->tisn); + } + write_lock_bh(&sk->sk_callback_lock); ulp_ddp_set_ctx(sk, NULL); write_unlock_bh(&sk->sk_callback_lock); diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.h b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.h index b9642e130b97..3bc45b81da06 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.h +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.h @@ -47,12 +47,16 @@ struct mlx5e_nvmeotcp_sq { * @sk: The socket used by the NVMe-TCP queue * @zerocopy: if this queue is used for zerocopy offload. * @crc_rx: if this queue is used for CRC Rx offload. + * @crc_tx: if this queue is used for CRC Tx offload. * @ccid: ID of the current CC * @ccsglidx: Index within the scatter-gather list (SGL) of the current CC * @ccoff_inner: Current offset within the @ccsglidx element * @priv: mlx5e netdev priv * @inv_done: invalidate callback of the nvme tcp driver * @after_resync_cqe: indicate if resync occurred + * @tisn: Destination TIS number created for NVMEoTCP CRC TX offload + * @pending: indicate if static/progress params need to be send to NIC. + * @end_seq_hint: Tx ooo - offload packet only if it ends after the hint. */ struct mlx5e_nvmeotcp_queue { struct ulp_ddp_ctx ulp_ddp_ctx; @@ -66,7 +70,7 @@ struct mlx5e_nvmeotcp_queue { u32 tag_buf_table_id; struct rhash_head hash; refcount_t ref_count; - bool dgst; + int dgst; int pda; u32 ccid_gen; u32 max_klms_per_wqe; @@ -74,6 +78,12 @@ struct mlx5e_nvmeotcp_queue { struct sock *sk; bool zerocopy; bool crc_rx; + bool crc_tx; + /* for crc_tx offload */ + int tisn; + bool pending; + u32 end_seq_hint; + u32 start_pdu_hint; /* current ccid fields */ off_t ccoff; From patchwork Thu Jul 22 11:03:21 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Boris Pismenny X-Patchwork-Id: 484310 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-20.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER, INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 705CDC6377D for ; Thu, 22 Jul 2021 11:06:58 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 556BC6120D for ; Thu, 22 Jul 2021 11:06:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231670AbhGVK0W (ORCPT ); Thu, 22 Jul 2021 06:26:22 -0400 Received: from mail-dm6nam11on2060.outbound.protection.outlook.com ([40.107.223.60]:5409 "EHLO NAM11-DM6-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S231634AbhGVK0U (ORCPT ); Thu, 22 Jul 2021 06:26:20 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Xv3txadA6JLUb/jtsDplnK4ut7rrpc7hQF+/rIknn8cN5sNdiw40HXF6r0OiAigymjhH61OaTqi5Od5LM7Il/TcgBHzeLVw7C7ph0d8Jj4rlNsvbMDTpLeTPzD79M2bEYxodCih8024z0hE6ZAbbSVZiBQYhdmOj4eQpqXrq3SkjUBjqL9DKC/AfFzYhy7V36CPAOPscaavKaTfeA0ZMdcAgNGm4PAPoLqldcAUmkjParbGwbbQslm0C3elNByOSmxZs6vrEohEhc7RfSvEw5c6lJJhUvXfMAsUZt5seqenCyMKZaqynmzk08aukvLptfxcatAyv//26ddkakOtx5A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=BWC1042/NGaBW9CLIxVx6DTXYRtNGfao542cVCSd1jk=; b=VGuWjefNLJBHSONfBRAc6qwpC/z3A3L2LejvCT5qEX5G7NOvXLoBXoyFx80Pk7tZRQMXBzR3tIeD6/FCzrVLySGhdu/N3MAJP0iFsHT0RIwRQj/tmoerxIE4AmKOxowSW7giniayLEilpnFCDEqsE6cE4sXP62fP7XobkAdtncm8db93mmEOi0LRrYo2iRgoQtE7+v7AjNgljOj0qLf+BxUYjZ90NXYtiakMuf/wR0rKTUwk+MoyHlvNzzdsZi+YOWJ6JFaf0+9EoQLfZbVAiLigfngQt9j8TMOkeQDNed/9qhZWdMPXu23eM35CMIUECMQPJWbdzirHAxpg8k4/QQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.36) smtp.rcpttodomain=marvell.com smtp.mailfrom=nvidia.com; dmarc=pass (p=quarantine sp=none pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=BWC1042/NGaBW9CLIxVx6DTXYRtNGfao542cVCSd1jk=; b=mS8hqJy3j7PTgcWD1BjfbKnpnnbO7yfGTqO2Y4k37/10+/RCGAnCJqaZ82hVKkcn+FwBP+z4Qn0KebFl6MWdhttg4ya7NxyMu41ORgZM5Re3Z2s7k/fmodcAe6qhwKbM8gEIUfEgjC4W5dFAGOOgh9LxaZa4khjQw0sabwO6yZqw0FgFSaS7TcocNvHKMoZXM3SlKOK5qgHPQnRzPUTaIgh6SdEgm8qfmlFIAoDTglB1j9j2dq0eXQwV3UPV7sBPAHWLh5JYg9cpnxyeD9WvpFeP4qMXTW46I3pjcZQxzVm4dTdz9Ld+sATqKBwH+jlzHHYqA00YOb2PNwbkEifH5w== Received: from MWHPR2001CA0007.namprd20.prod.outlook.com (2603:10b6:301:15::17) by MWHPR1201MB0157.namprd12.prod.outlook.com (2603:10b6:301:55::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4331.30; Thu, 22 Jul 2021 11:06:53 +0000 Received: from CO1NAM11FT038.eop-nam11.prod.protection.outlook.com (2603:10b6:301:15:cafe::60) by MWHPR2001CA0007.outlook.office365.com (2603:10b6:301:15::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4352.25 via Frontend Transport; Thu, 22 Jul 2021 11:06:53 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.36) smtp.mailfrom=nvidia.com; marvell.com; dkim=none (message not signed) header.d=none; marvell.com; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.36 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.36; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.36) by CO1NAM11FT038.mail.protection.outlook.com (10.13.174.231) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4352.24 via Frontend Transport; Thu, 22 Jul 2021 11:06:53 +0000 Received: from HQMAIL107.nvidia.com (172.20.187.13) by HQMAIL101.nvidia.com (172.20.187.10) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Thu, 22 Jul 2021 11:06:52 +0000 Received: from vdi.nvidia.com (172.20.187.5) by mail.nvidia.com (172.20.187.13) with Microsoft SMTP Server id 15.0.1497.2 via Frontend Transport; Thu, 22 Jul 2021 11:06:47 +0000 From: Boris Pismenny To: , , , , , , , , , , CC: , , , , , Subject: [PATCH v5 net-next 32/36] net/mlx5e: NVMEoTCP DDGST TX BSF and PSV Date: Thu, 22 Jul 2021 14:03:21 +0300 Message-ID: <20210722110325.371-33-borisp@nvidia.com> X-Mailer: git-send-email 2.24.1 In-Reply-To: <20210722110325.371-1-borisp@nvidia.com> References: <20210722110325.371-1-borisp@nvidia.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: e8a2e30b-5da2-444f-b099-08d94d00d01a X-MS-TrafficTypeDiagnostic: MWHPR1201MB0157: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:2089; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: epFaiSZBusXF0BexeSDABq3jlJU2j2ITe0/lnAmkYAucdzeYdzwUsLPRejlCWm+7b6l15NT7tmk+4h5pzI0z5N2L7DgzsxC4ENst823kolHPlYoKZWqlTcAggOdHS8fLpXJigzOMxBO4fN0efGF0el+v2Xesy7XfEDTGJMlDqnLMYcxZEZ3ReVZPHglxnrKS5tM+PR+QyWVP7vMF6Lyr5ldD2cAiVwDtNWoH677o9PdFhNdlaIvpmr81BqST14ko4IF2jBNeu8vkmyFI8A8zlJ06RACKus6K8rfCc2cQyE9A+gy1uYyhwSOKs8w6gDTw/Kp7HmsE9yH0JpZG5bi6TXxL7NvQv84cGknxtm1dwdeGoHd+05bOoAePVdVzC+PzKite2Kretfg61JgrLR0JG/66OhNFYgjS+dWd9Gk7Zrjo9ByEWMytqrH5sX84S4NxSxcOYKgnHV4gOxqzwt5/D8WDQLilA7i+0TtH6LurxmCKZ/ELWFHWK5f0oPXA5xLrdZcWBSZoAMMSMxnv+go5O0wLdnYXMDPJUowVHQIOGiou9YRfJ8bTW8sRNE+UjyNOhiivdgxtsWc78jDTzlYqjQWhXEOXQJs+70kuofd4F1i+QQ/oO2NbCWCevQjtydgxsV1fUGqQveTFSLPrXY2e13C/ZjC4nX/gbEC2vRuQCECJqRK4kK2xiyXtWQloDakCMQcTarckNe4M56v+vJ03FOmwqpzE5zEh+PoQhbM2VAs= X-Forefront-Antispam-Report: CIP:216.228.112.36; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:schybrid05.nvidia.com; CAT:NONE; SFS:(4636009)(376002)(396003)(346002)(39860400002)(136003)(46966006)(36840700001)(70206006)(70586007)(107886003)(5660300002)(7416002)(82310400003)(7636003)(8936002)(54906003)(36860700001)(4326008)(36906005)(478600001)(356005)(316002)(110136005)(82740400003)(186003)(47076005)(83380400001)(336012)(36756003)(2906002)(6666004)(921005)(8676002)(2616005)(26005)(86362001)(426003)(7696005)(1076003); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Jul 2021 11:06:53.4505 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: e8a2e30b-5da2-444f-b099-08d94d00d01a X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.112.36]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CO1NAM11FT038.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR1201MB0157 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Yoray Zack Change the function that build NVMEoTCP progress params and static params, to work for Tx/Rx. Signed-off-by: Yoray Zack --- .../mellanox/mlx5/core/en_accel/nvmeotcp.c | 130 ++++++++++++++---- .../mlx5/core/en_accel/nvmeotcp_utils.h | 4 +- 2 files changed, 108 insertions(+), 26 deletions(-) diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c index 6023e1ae7be4..624d8a28dc21 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c @@ -155,8 +155,11 @@ static int mlx5e_nvmeotcp_create_tis(struct mlx5_core_dev *mdev, u32 *tisn) } #define MLX5_CTRL_SEGMENT_OPC_MOD_UMR_TIR_PARAMS 0x2 +#define MLX5_CTRL_SEGMENT_OPC_MOD_UMR_TIS_PARAMS 0x1 #define MLX5_CTRL_SEGMENT_OPC_MOD_UMR_NVMEOTCP_TIR_STATIC_PARAMS 0x2 #define MLX5_CTRL_SEGMENT_OPC_MOD_UMR_UMR 0x0 +#define MLX5_CTRL_SEGMENT_OPC_MOD_UMR_NVMEOTCP_TIS_STATIC_PARAMS 0x1 +#define MLX5_CTRL_SEGMENT_OPC_MOD_UMR_NVMEOTCP_TIS_PROGRESS_PARAMS 0x3 #define STATIC_PARAMS_DS_CNT \ DIV_ROUND_UP(MLX5E_NVMEOTCP_STATIC_PARAMS_WQE_SZ, MLX5_SEND_WQE_DS) @@ -250,56 +253,75 @@ build_nvmeotcp_klm_umr(struct mlx5e_nvmeotcp_queue *queue, static void fill_nvmeotcp_progress_params(struct mlx5e_nvmeotcp_queue *queue, struct mlx5_seg_nvmeotcp_progress_params *params, - u32 seq) + u32 seq, bool is_tx) { void *ctx = params->ctx; - params->tir_num = cpu_to_be32(queue->tirn); + params->tir_num = is_tx ? cpu_to_be32(queue->tisn) : cpu_to_be32(queue->tirn); MLX5_SET(nvmeotcp_progress_params, ctx, next_pdu_tcp_sn, seq); MLX5_SET(nvmeotcp_progress_params, ctx, pdu_tracker_state, MLX5E_NVMEOTCP_PROGRESS_PARAMS_PDU_TRACKER_STATE_START); + if (is_tx) + MLX5_SET(nvmeotcp_progress_params, ctx, offloading_state, 0); +} + +static void nvme_tx_fill_wi(struct mlx5e_txqsq *sq, + u16 pi, u8 num_wqebbs, u32 num_bytes, + struct page *page, enum mlx5e_dump_wqe_type type) +{ + struct mlx5e_tx_wqe_info *wi = &sq->db.wqe_info[pi]; + + *wi = (struct mlx5e_tx_wqe_info) { + .num_wqebbs = num_wqebbs, + .num_bytes = num_bytes, + }; } void build_nvmeotcp_progress_params(struct mlx5e_nvmeotcp_queue *queue, struct mlx5e_set_nvmeotcp_progress_params_wqe *wqe, - u32 seq) + u32 seq, bool is_rx, bool resync, u16 pc, u32 sqn) { struct mlx5_wqe_ctrl_seg *cseg = &wqe->ctrl; - u32 sqn = queue->sq->icosq.sqn; - u16 pc = queue->sq->icosq.pc; - u8 opc_mod; + u8 opc_mod = is_rx ? + MLX5_CTRL_SEGMENT_OPC_MOD_UMR_NVMEOTCP_TIR_PROGRESS_PARAMS : + MLX5_CTRL_SEGMENT_OPC_MOD_UMR_NVMEOTCP_TIS_PROGRESS_PARAMS; memset(wqe, 0, MLX5E_NVMEOTCP_PROGRESS_PARAMS_WQE_SZ); - opc_mod = MLX5_CTRL_SEGMENT_OPC_MOD_UMR_NVMEOTCP_TIR_PROGRESS_PARAMS; + cseg->opmod_idx_opcode = cpu_to_be32((pc << MLX5_WQE_CTRL_WQE_INDEX_SHIFT) | MLX5_OPCODE_SET_PSV | (opc_mod << 24)); cseg->qpn_ds = cpu_to_be32((sqn << MLX5_WQE_CTRL_QPN_SHIFT) | PROGRESS_PARAMS_DS_CNT); - fill_nvmeotcp_progress_params(queue, &wqe->params, seq); + fill_nvmeotcp_progress_params(queue, &wqe->params, seq, !is_rx); } static void fill_nvmeotcp_static_params(struct mlx5e_nvmeotcp_queue *queue, struct mlx5_seg_nvmeotcp_static_params *params, - u32 resync_seq, bool zero_copy_en, + u32 resync_seq, bool is_rx, bool zero_copy_en, bool ddgst_offload_en) { void *ctx = params->ctx; + int pda = queue->pda; + bool hddgst_en = queue->dgst & NVME_TCP_HDR_DIGEST_ENABLE; + bool ddgst_en = queue->dgst & NVME_TCP_DATA_DIGEST_ENABLE; + + if (!is_rx) { + pda = 0; + } MLX5_SET(transport_static_params, ctx, const_1, 1); MLX5_SET(transport_static_params, ctx, const_2, 2); MLX5_SET(transport_static_params, ctx, acc_type, MLX5_TRANSPORT_STATIC_PARAMS_ACC_TYPE_NVMETCP); MLX5_SET(transport_static_params, ctx, nvme_resync_tcp_sn, resync_seq); - MLX5_SET(transport_static_params, ctx, pda, queue->pda); - MLX5_SET(transport_static_params, ctx, ddgst_en, - queue->dgst & NVME_TCP_DATA_DIGEST_ENABLE); + MLX5_SET(transport_static_params, ctx, pda, pda); + MLX5_SET(transport_static_params, ctx, ddgst_en, ddgst_en); MLX5_SET(transport_static_params, ctx, ddgst_offload_en, ddgst_offload_en); - MLX5_SET(transport_static_params, ctx, hddgst_en, - queue->dgst & NVME_TCP_HDR_DIGEST_ENABLE); + MLX5_SET(transport_static_params, ctx, hddgst_en, hddgst_en); MLX5_SET(transport_static_params, ctx, hdgst_offload_en, 0); MLX5_SET(transport_static_params, ctx, ti, MLX5_TRANSPORT_STATIC_PARAMS_TI_INITIATOR); @@ -310,26 +332,31 @@ fill_nvmeotcp_static_params(struct mlx5e_nvmeotcp_queue *queue, void build_nvmeotcp_static_params(struct mlx5e_nvmeotcp_queue *queue, struct mlx5e_set_nvmeotcp_static_params_wqe *wqe, - u32 resync_seq, bool zerocopy, bool crc_rx) + u32 resync_seq, bool is_rx, u16 pc, u32 sqn, + bool zerocopy, bool crc_rx) { - u8 opc_mod = MLX5_CTRL_SEGMENT_OPC_MOD_UMR_NVMEOTCP_TIR_STATIC_PARAMS; struct mlx5_wqe_umr_ctrl_seg *ucseg = &wqe->uctrl; struct mlx5_wqe_ctrl_seg *cseg = &wqe->ctrl; - u32 sqn = queue->sq->icosq.sqn; - u16 pc = queue->sq->icosq.pc; + int tirn_tisn = is_rx ? queue->tirn : queue->tisn; + u8 opc_mod = is_rx ? + MLX5_CTRL_SEGMENT_OPC_MOD_UMR_NVMEOTCP_TIR_STATIC_PARAMS : + MLX5_CTRL_SEGMENT_OPC_MOD_UMR_NVMEOTCP_TIS_STATIC_PARAMS; + memset(wqe, 0, MLX5E_NVMEOTCP_STATIC_PARAMS_WQE_SZ); - cseg->opmod_idx_opcode = cpu_to_be32((pc << MLX5_WQE_CTRL_WQE_INDEX_SHIFT) | - MLX5_OPCODE_UMR | (opc_mod) << 24); + cseg->opmod_idx_opcode = cpu_to_be32((pc << MLX5_WQE_CTRL_WQE_INDEX_SHIFT) | + MLX5_OPCODE_UMR | (opc_mod) << 24); cseg->qpn_ds = cpu_to_be32((sqn << MLX5_WQE_CTRL_QPN_SHIFT) | STATIC_PARAMS_DS_CNT); - cseg->imm = cpu_to_be32(queue->tirn << MLX5_WQE_CTRL_TIR_TIS_INDEX_SHIFT); + cseg->imm = cpu_to_be32(tirn_tisn << + MLX5_WQE_CTRL_TIR_TIS_INDEX_SHIFT); ucseg->flags = MLX5_UMR_INLINE; ucseg->bsf_octowords = cpu_to_be16(MLX5E_NVMEOTCP_STATIC_PARAMS_OCTWORD_SIZE); - fill_nvmeotcp_static_params(queue, &wqe->params, resync_seq, zerocopy, crc_rx); + fill_nvmeotcp_static_params(queue, &wqe->params, resync_seq, + is_rx, zerocopy, crc_rx); } static void @@ -371,7 +398,8 @@ mlx5e_nvmeotcp_rx_post_static_params_wqe(struct mlx5e_nvmeotcp_queue *queue, pi = mlx5e_icosq_get_next_pi(sq, wqe_bbs); wqe = MLX5E_NVMEOTCP_FETCH_STATIC_PARAMS_WQE(sq, pi); mlx5e_nvmeotcp_fill_wi(NULL, sq, wqe_bbs, pi, 0, BSF_UMR); - build_nvmeotcp_static_params(queue, wqe, resync_seq, queue->zerocopy, queue->crc_rx); + build_nvmeotcp_static_params(queue, wqe, resync_seq, true, sq->pc, + sq->sqn, queue->zerocopy, queue->crc_rx); sq->pc += wqe_bbs; mlx5e_notify_hw(&sq->wq, sq->pc, sq->uar_map, &wqe->ctrl); spin_unlock(&queue->nvmeotcp_icosq_lock); @@ -389,7 +417,7 @@ mlx5e_nvmeotcp_rx_post_progress_params_wqe(struct mlx5e_nvmeotcp_queue *queue, pi = mlx5e_icosq_get_next_pi(sq, wqe_bbs); wqe = MLX5E_NVMEOTCP_FETCH_PROGRESS_PARAMS_WQE(sq, pi); mlx5e_nvmeotcp_fill_wi(queue, sq, wqe_bbs, pi, 0, SET_PSV_UMR); - build_nvmeotcp_progress_params(queue, wqe, seq); + build_nvmeotcp_progress_params(queue, wqe, seq, true, false, sq->pc, sq->sqn); sq->pc += wqe_bbs; mlx5e_notify_hw(&sq->wq, sq->pc, sq->uar_map, &wqe->ctrl); } @@ -1078,6 +1106,60 @@ int mlx5e_nvmeotcp_init(struct mlx5e_priv *priv) return ret; } +static +void mlx5e_nvmeotcp_tx_post_static_params(struct mlx5e_nvmeotcp_queue *queue, + struct mlx5e_txqsq *sq) +{ + struct mlx5e_set_nvmeotcp_static_params_wqe *wqe; + enum mlx5e_dump_wqe_type type = MLX5E_DUMP_WQE_NVMEOTCP; + u16 pi, wqe_bbs; + + wqe_bbs = MLX5E_NVMEOTCP_STATIC_PARAMS_WQEBBS; + pi = mlx5e_txqsq_get_next_pi(sq, wqe_bbs); + wqe = MLX5E_NVMEOTCP_FETCH_STATIC_PARAMS_WQE(sq, pi); + nvme_tx_fill_wi(sq, pi, wqe_bbs, 0, NULL, type); + build_nvmeotcp_static_params(queue, wqe, 0, false, + sq->pc, sq->sqn, false, true); + sq->pc += wqe_bbs; + mlx5e_notify_hw(&sq->wq, sq->pc, sq->uar_map, &wqe->ctrl); +} + +static +void mlx5e_nvmeotcp_tx_post_progress_params(struct mlx5e_nvmeotcp_queue *queue, + struct mlx5e_txqsq *sq, u32 seq, + bool resync) +{ + struct mlx5e_set_nvmeotcp_progress_params_wqe *wqe; + enum mlx5e_dump_wqe_type type = MLX5E_DUMP_WQE_NVMEOTCP; + u16 pi, wqe_bbs; + + wqe_bbs = MLX5E_NVMEOTCP_PROGRESS_PARAMS_WQEBBS; + pi = mlx5e_txqsq_get_next_pi(sq, wqe_bbs); + wqe = MLX5E_NVMEOTCP_FETCH_PROGRESS_PARAMS_WQE(sq, pi); + nvme_tx_fill_wi(sq, pi, wqe_bbs, 0, NULL, type); + build_nvmeotcp_progress_params(queue, wqe, seq, false, resync, sq->pc, sq->sqn); + sq->pc += wqe_bbs; + mlx5e_notify_hw(&sq->wq, sq->pc, sq->uar_map, &wqe->ctrl); +} + +static +bool mlx5e_nvmeotcp_test_and_clear_pending(struct mlx5e_nvmeotcp_queue *ctx) +{ + bool ret = ctx->pending; + + ctx->pending = false; + + return ret; +} + +static +void mlx5e_nvmeotcp_tx_post_param_wqes(struct mlx5e_txqsq *sq, struct sock *sk, + struct mlx5e_nvmeotcp_queue *ctx) +{ + mlx5e_nvmeotcp_tx_post_static_params(ctx, sq); + mlx5e_nvmeotcp_tx_post_progress_params(ctx, sq, tcp_sk(sk)->copied_seq, false); +} + void mlx5e_nvmeotcp_cleanup(struct mlx5e_priv *priv) { struct mlx5e_nvmeotcp *nvmeotcp = priv->nvmeotcp; diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp_utils.h b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp_utils.h index 44671e28a9ea..e7436aa01ad4 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp_utils.h +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp_utils.h @@ -69,12 +69,12 @@ struct mlx5e_get_psv_wqe { void build_nvmeotcp_progress_params(struct mlx5e_nvmeotcp_queue *queue, struct mlx5e_set_nvmeotcp_progress_params_wqe *wqe, - u32 seq); + u32 seq, bool is_rx, bool is_resync, u16 pc, u32 sqn); void build_nvmeotcp_static_params(struct mlx5e_nvmeotcp_queue *queue, struct mlx5e_set_nvmeotcp_static_params_wqe *wqe, - u32 resync_seq, + u32 resync_seq, bool is_rx, u16 pc, u32 sqn, bool zerocopy, bool crc_rx); #endif /* __MLX5E_NVMEOTCP_UTILS_H__ */ From patchwork Thu Jul 22 11:03:22 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Boris Pismenny X-Patchwork-Id: 485096 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-20.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER, INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5A6F0C63793 for ; Thu, 22 Jul 2021 11:07:06 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 401A06120D for ; Thu, 22 Jul 2021 11:07:06 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231830AbhGVK0a (ORCPT ); Thu, 22 Jul 2021 06:26:30 -0400 Received: from mail-co1nam11on2070.outbound.protection.outlook.com ([40.107.220.70]:10720 "EHLO NAM11-CO1-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S231634AbhGVK02 (ORCPT ); Thu, 22 Jul 2021 06:26:28 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=WMFvjXtNxUKb/ntFUq3I+fES2Wtg/+XXClZxW8RJE85NZryrEjX6k0XcXBRWujwRNp09+oHs1jdgDqruRHsr0YHu7MSC94Mz8+Z1sUoob4JKT5EjugSoVnvvBifeh25Ly1LENmoYw3iYtAsDKP/W8DTIXKwzBj3sEtzXFGw+0l7uBW/UvXn5fYrLncwQlvDvyeDbAceNy53fnKhb8708kn0RtKcg1Cryn2lwmxRdDE/nzfVO5auVrbqjcH14EKEpkJ+dM58cYwttCCgajr1KgDxl3UQ8mQVtAdZGGtMfZf5c9TfvMRBHOFMuhSVjmEdJc/dmlBu4Q/XGladPwrPzFg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=1h5UfwDtV1F1ZrdJ5wPNVZcWPRuJLVii37OhhJ80aX4=; b=Bq9P3ZUTKZVZM0/O+wEsxFQcDt3EOdbNNs8aAHmRBzS7ejD7V+jTmHTqqdzVYcRtYIvvNkvXiCRqavvKg0U3zAOOBMBa63VUZ7Vdq7Izx99MwdTsoL1uXywClItAJ73oChc0UXS41NQRmGjsA/zDz58ABl/b5M3ENlpN0OT1C23LPZZdIlfjkyd8JFYhS2WB+cLsQGtL683KPnBEAWC0/9s3QjpA00J34NOtZDluj8FKhT9ZmGpoUIbBRq4/nc4H49apRL/BHZ4n2WdzLGuUEPEFo9xV2NSDcx2Jl8SMb76W6qi8oNIUz3ob0sDSceFlHBSRc6UWvT4k3wz3exc5aQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.36) smtp.rcpttodomain=marvell.com smtp.mailfrom=nvidia.com; dmarc=pass (p=quarantine sp=none pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=1h5UfwDtV1F1ZrdJ5wPNVZcWPRuJLVii37OhhJ80aX4=; b=b1Oxe11GNADnNYcxIFsGUvPF+g21hSdWedRGWxxAShc64p+vjjKGN2I76yul1FM4xXGcXNxjXhI57ZNwVRGQUR7tkFe7Xv5hKBi/M5rOr/KbwRF1iBK93HslGypdeceOBmT7oiWzxrYo6W+GHHKOoW/hgWilAZxZO/z6VnEWvutJ79ucDtJAX9+GK4HFPoP8nTl6qkUXtGluZMuq0cQZkyun8EsDmYSknKWxowgsaLHeDnkKQZAsP0Y5C2hpPv1I1FKK4AhuaZNWry7twznCADxDV4O3kQLASk3Tx/vqEOZFEjTAmsazleIv+GNt/IZqkGR2QsL/2ewroXTSpCTfNQ== Received: from MWHPR17CA0069.namprd17.prod.outlook.com (2603:10b6:300:93::31) by MN2PR12MB3405.namprd12.prod.outlook.com (2603:10b6:208:cb::32) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4331.29; Thu, 22 Jul 2021 11:07:02 +0000 Received: from CO1NAM11FT027.eop-nam11.prod.protection.outlook.com (2603:10b6:300:93:cafe::73) by MWHPR17CA0069.outlook.office365.com (2603:10b6:300:93::31) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4352.25 via Frontend Transport; Thu, 22 Jul 2021 11:07:01 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.36) smtp.mailfrom=nvidia.com; marvell.com; dkim=none (message not signed) header.d=none; marvell.com; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.36 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.36; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.36) by CO1NAM11FT027.mail.protection.outlook.com (10.13.174.224) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4352.24 via Frontend Transport; Thu, 22 Jul 2021 11:07:01 +0000 Received: from HQMAIL107.nvidia.com (172.20.187.13) by HQMAIL101.nvidia.com (172.20.187.10) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Thu, 22 Jul 2021 11:07:00 +0000 Received: from vdi.nvidia.com (172.20.187.5) by mail.nvidia.com (172.20.187.13) with Microsoft SMTP Server id 15.0.1497.2 via Frontend Transport; Thu, 22 Jul 2021 11:06:53 +0000 From: Boris Pismenny To: , , , , , , , , , , CC: , , , , , Subject: [PATCH v5 net-next 33/36] net/mlx5e: NVMEoTCP DDGST TX Data path Date: Thu, 22 Jul 2021 14:03:22 +0300 Message-ID: <20210722110325.371-34-borisp@nvidia.com> X-Mailer: git-send-email 2.24.1 In-Reply-To: <20210722110325.371-1-borisp@nvidia.com> References: <20210722110325.371-1-borisp@nvidia.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: a32b389e-cef2-4980-cca6-08d94d00d4c7 X-MS-TrafficTypeDiagnostic: MN2PR12MB3405: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:6108; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: fLAGFQS60QpNuPf+LufZjDzqU04sn33xOCmNETbt4wMj4v4l+c6yQILLqpW387StsAr4XHFdUzx3fsvbxXGWl7YNT/3ELgA7JJASbUqt7YNrQ/FwNrXnmElqallJgL7lcjGQ23jPjtbbhCAoWP94Hee5dUdLaIqTBU9IJ+uhIaqnv/W3gV6qLyBzXM+ouTa9B9J6q4oK5sXHbEkYrmRChgfo5UY966RgKRLtNpGpcXYzVbDl5mBMQ3TkT2ZqG7xwP3I8hnr9BQu7jMy4QgoMM8vOLjXxOI6kaLNyudQ4Ka7Z+bKjVX6nyyh9QfvFrTV34DzqHKzYXtCnRjd9ik01sObpXFPtoL0XBcafBBcCtJvLPYZ8qpZtMZlHnap4wJ7/vyU3g+46mYOrOSTUR1pYxzKCJpVA78mqSr+dbmuS7UbAC2TVbG6xY7XsMHPw8NZL6j5JESs8eNMfr2w560yR+i5pPGUzWeavluLYz+QQRM1IBU9IkeC3Q0LSb6OU3VJEdWm15qFBvWxsfypowmlXS3vR2/ZpeGPwLhIEUziA+yXVcfHuCopikJ4FJWN6CsGKZwBny1x/4UaFTY97JcqYpWPSg+qki+WQ7BOBxvjlrQcBGR/LzccMf0PIO9djHjBOYMVC+GV8qkWM7qw1OFATSqEGuMCKvf3X9OlWWP16DM5bvGJNuGCoeghDl+phrbLluZrwwoLzEkPl5m+uuLXJ11up8CBOsa4xViotLM3Sht0= X-Forefront-Antispam-Report: CIP:216.228.112.36; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:schybrid05.nvidia.com; CAT:NONE; SFS:(4636009)(36840700001)(46966006)(186003)(70206006)(4326008)(6666004)(47076005)(336012)(7416002)(921005)(82310400003)(2906002)(5660300002)(2616005)(107886003)(426003)(7636003)(356005)(70586007)(7696005)(1076003)(26005)(508600001)(83380400001)(36756003)(8676002)(316002)(86362001)(110136005)(36860700001)(8936002)(54906003)(36906005); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Jul 2021 11:07:01.3739 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: a32b389e-cef2-4980-cca6-08d94d00d4c7 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.112.36]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CO1NAM11FT027.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR12MB3405 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Yoray Zack This patch handles only the good flow for the DDGST Tx offload skb and wqe. Later patches will handle bad flow (OOO packets) 1. add mlx5e_nvmeotcp_handle_tx_skb function(skb,...): check if the skb can be offlaoded. this function track the tcp_seq of the skb, and check if this is the next tcp_seq. and if so, send this skb with DDGST Tx offload. 2. add mlx5e_nvmeotcp_handle_tx_wqe function : send the wqe with the correct tis number for the offload. Signed-off-by: Yoray Zack --- .../mellanox/mlx5/core/en_accel/en_accel.h | 13 ++++ .../mellanox/mlx5/core/en_accel/nvmeotcp.c | 63 +++++++++++++++++++ .../mellanox/mlx5/core/en_accel/nvmeotcp.h | 6 ++ .../net/ethernet/mellanox/mlx5/core/en_tx.c | 11 ++++ 4 files changed, 93 insertions(+) diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/en_accel.h b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/en_accel.h index b9404366e6e8..2e5a7741736f 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/en_accel.h +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/en_accel.h @@ -113,6 +113,9 @@ struct mlx5e_accel_tx_state { #ifdef CONFIG_MLX5_EN_IPSEC struct mlx5e_accel_tx_ipsec_state ipsec; #endif +#ifdef CONFIG_MLX5_EN_NVMEOTCP + int tisn; +#endif }; static inline bool mlx5e_accel_tx_begin(struct net_device *dev, @@ -137,6 +140,12 @@ static inline bool mlx5e_accel_tx_begin(struct net_device *dev, } #endif +#ifdef CONFIG_MLX5_EN_NVMEOTCP + if (test_bit(MLX5E_SQ_STATE_NVMEOTCP, &sq->state)) { + if (unlikely(!mlx5e_nvmeotcp_handle_tx_skb(dev, sq, skb, &state->tisn))) + return false; + } +#endif return true; } @@ -187,6 +196,10 @@ static inline void mlx5e_accel_tx_finish(struct mlx5e_txqsq *sq, mlx5e_tls_handle_tx_wqe(&wqe->ctrl, &state->tls); #endif +#ifdef CONFIG_MLX5_EN_NVMEOTCP + mlx5e_nvmeotcp_handle_tx_wqe(sq, &wqe->ctrl, state->tisn); +#endif + #ifdef CONFIG_MLX5_EN_IPSEC if (test_bit(MLX5E_SQ_STATE_IPSEC, &sq->state) && state->ipsec.xo && state->ipsec.tailen) diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c index 624d8a28dc21..d9f6125f5dbc 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c @@ -1160,6 +1160,69 @@ void mlx5e_nvmeotcp_tx_post_param_wqes(struct mlx5e_txqsq *sq, struct sock *sk, mlx5e_nvmeotcp_tx_post_progress_params(ctx, sq, tcp_sk(sk)->copied_seq, false); } +static inline bool mlx5e_is_sk_tx_device_offloaded(struct sock *sk) +{ + /* Return True after smp_store_release assing in + * mlx5e_nvmeotcp_queue_tx_init(). + */ + return sk && sk_fullsock(sk) && + (smp_load_acquire(&sk->sk_validate_xmit_skb) == + &ulp_ddp_validate_xmit_skb); +} + +bool mlx5e_nvmeotcp_handle_tx_skb(struct net_device *netdev, + struct mlx5e_txqsq *sq, + struct sk_buff *skb, int *nvmeotcp_tisn) +{ + struct mlx5e_nvmeotcp_queue *ctx; + int datalen; + u32 seq; + + if (!mlx5e_is_sk_tx_device_offloaded(skb->sk)) + goto out; + + datalen = skb->len - (skb_transport_offset(skb) + tcp_hdrlen(skb)); + if (!datalen) + goto out; + + ctx = container_of(ulp_ddp_get_ctx(skb->sk), + struct mlx5e_nvmeotcp_queue, ulp_ddp_ctx); + if (!ctx) + goto out; + + mlx5e_tx_mpwqe_ensure_complete(sq); + + if (WARN_ON_ONCE(ctx->ulp_ddp_ctx.netdev != netdev)) + goto err_out; + + if (unlikely(mlx5e_nvmeotcp_test_and_clear_pending(ctx))) + mlx5e_nvmeotcp_tx_post_param_wqes(sq, skb->sk, ctx); + + seq = ntohl(tcp_hdr(skb)->seq); + if (unlikely(ctx->ulp_ddp_ctx.expected_seq != seq)) + goto err_out; + + *nvmeotcp_tisn = ctx->tisn; + ctx->ulp_ddp_ctx.expected_seq = seq + datalen; + goto good_out; +out: + *nvmeotcp_tisn = 0; +good_out: + return true; +err_out: + dev_kfree_skb(skb); + return false; +} + +void mlx5e_nvmeotcp_handle_tx_wqe(struct mlx5e_txqsq *sq, + struct mlx5_wqe_ctrl_seg *cseg, + int tisn) +{ + if (tisn) { + cseg->tis_tir_num = cpu_to_be32(tisn << 8); + } +} + void mlx5e_nvmeotcp_cleanup(struct mlx5e_priv *priv) { struct mlx5e_nvmeotcp *nvmeotcp = priv->nvmeotcp; diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.h b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.h index 3bc45b81da06..0451b3ac3687 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.h +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.h @@ -119,6 +119,12 @@ void mlx5e_nvmeotcp_ddp_inv_done(struct mlx5e_icosq_wqe_info *wi); void mlx5e_nvmeotcp_ctx_comp(struct mlx5e_icosq_wqe_info *wi); int mlx5e_nvmeotcp_init_rx(struct mlx5e_priv *priv); void mlx5e_nvmeotcp_cleanup_rx(struct mlx5e_priv *priv); +bool mlx5e_nvmeotcp_handle_tx_skb(struct net_device *netdev, + struct mlx5e_txqsq *sq, + struct sk_buff *skb, int *tisn); +void mlx5e_nvmeotcp_handle_tx_wqe(struct mlx5e_txqsq *sq, + struct mlx5_wqe_ctrl_seg *csegl, + int tisn); #else static inline void mlx5e_nvmeotcp_build_netdev(struct mlx5e_priv *priv) { } diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_tx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_tx.c index c63d78eda606..7feaf5608b9d 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_tx.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_tx.c @@ -249,6 +249,13 @@ mlx5e_txwqe_build_eseg_csum(struct mlx5e_txqsq *sq, struct sk_buff *skb, eseg->cs_flags = MLX5_ETH_WQE_L3_CSUM | MLX5_ETH_WQE_L4_CSUM; sq->stats->csum_partial++; #endif +#ifdef CONFIG_MLX5_EN_NVMEOTCP + } else if (unlikely(accel && accel->tisn)) { + eseg->cs_flags = MLX5_ETH_WQE_L3_CSUM | MLX5_ETH_WQE_L4_CSUM; + sq->stats->csum_partial++; +#endif + + } else if (unlikely(mlx5e_ipsec_eseg_meta(eseg))) { ipsec_txwqe_build_eseg_csum(sq, skb, eseg); } else @@ -352,6 +359,10 @@ mlx5e_tx_wqe_inline_mode(struct mlx5e_txqsq *sq, struct sk_buff *skb, if (accel && accel->tls.tls_tisn) return MLX5_INLINE_MODE_TCP_UDP; #endif +#ifdef CONFIG_MLX5_EN_NVMEOTCP + if (accel && accel->tisn) + return MLX5_INLINE_MODE_TCP_UDP; +#endif mode = sq->min_inline_mode; From patchwork Thu Jul 22 11:03:23 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Boris Pismenny X-Patchwork-Id: 484309 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-20.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER, INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1F5EBC63793 for ; Thu, 22 Jul 2021 11:07:13 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 08A8B6120D for ; Thu, 22 Jul 2021 11:07:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231843AbhGVK0h (ORCPT ); Thu, 22 Jul 2021 06:26:37 -0400 Received: from mail-dm6nam10on2070.outbound.protection.outlook.com ([40.107.93.70]:13857 "EHLO NAM10-DM6-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S231634AbhGVK0g (ORCPT ); Thu, 22 Jul 2021 06:26:36 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=TgpQGWDVc3QKCpPiF0+Kh8gPaOzXJymTWWZ6EJ4ki0JgXWTFGsNrxB6pdEsYo1mXcBUSQ75PSKrLjbMHD/Qfp8Ci//YyA5JYxj2ChzBisFNgQv/e/Za0BfhrF/FsTSSlW0luyDMx6F/b+gc5Y9FzvFBq1wzN/BNG1FxcqL1oPH4RKVaJwy5Vq1U+dy9TynvD33TxZDaasJ9oH2uC47H+rfT258t75sVVkmty4lpnP02YJ6WjpH5YYqgq0B01MQws5RGzE7nVoIJgBMJdDtbjSsUG7p2qJWNaWzBMb570GbOHv9PnyYRa0Cy8gS1YctWOGyodzlG3tmyQP/CasbkTiw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=ICNaHiF0X9S85BDGO3H6Y0pJMNddfvC8aK44dcQ8GFw=; b=ZX+IMVa3b6mkqFN6yQcu3i9cGA6gczV98iBhGjj+jgS8F1pTc8bjXH9k+PnxlZDgex85ivwmvY0G3FgWNoL/WMvxISrBO5HyC5TaS+xBqNFgar6+NopdT4dLFQodaZaZvEFmuvGGJxaOcU7P3jo9GAvkdYQ0atG5pEARDHIQOYoKMStsyiCHeEOEuG6Uxgq53R0PxZDTQQBQS3zfGpVh+aUOaH0Ao7+VmXlsncmkLrtGhQWpaNyKaASgyrOvWQGDckbTRx5KGftafvh5NFpU2WpIcwtJMcAWHGNOPsiKT0wUv0Modc1RosTGrV+pA7B/OOUPv430iTj+7LEGH8CG3w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.32) smtp.rcpttodomain=marvell.com smtp.mailfrom=nvidia.com; dmarc=pass (p=quarantine sp=none pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=ICNaHiF0X9S85BDGO3H6Y0pJMNddfvC8aK44dcQ8GFw=; b=Ej3sovYcjPAI4NhsCy7/cBgL7MyrcrBEYDAZuDbB48wKppGBsZmpThuJas7VElsB1WNGybZuCim8pWNL2pLgD8SAJKh6b8Ud3JQ+RXmbjryetqhT9BJD084m2QJuKNfKtdpZino0eBigMtKeaLQUPqPBUDtFD9uXw8ir4H+YsAiWnLbDiW+s5WBLz2WuoLm3gMsC19LEmpxCum95Mt/8qx0EQGgOQoNvyuuF0qwzjQBSfab9EXdAl6jw3UpGneuJ24sxYKQCAjBp4Iecvy+fW2hNp0qq+SYw4ds++GWdjPZwoFKZEjvDWmgs+OYgY2WyrUw1eeavId7/QIW0NnM50A== Received: from DM5PR13CA0022.namprd13.prod.outlook.com (2603:10b6:3:23::32) by MN2PR12MB4015.namprd12.prod.outlook.com (2603:10b6:208:16e::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4331.26; Thu, 22 Jul 2021 11:07:09 +0000 Received: from DM6NAM11FT013.eop-nam11.prod.protection.outlook.com (2603:10b6:3:23:cafe::36) by DM5PR13CA0022.outlook.office365.com (2603:10b6:3:23::32) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4373.7 via Frontend Transport; Thu, 22 Jul 2021 11:07:09 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.32) smtp.mailfrom=nvidia.com; marvell.com; dkim=none (message not signed) header.d=none; marvell.com; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.32 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.32; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.32) by DM6NAM11FT013.mail.protection.outlook.com (10.13.173.142) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4352.24 via Frontend Transport; Thu, 22 Jul 2021 11:07:09 +0000 Received: from HQMAIL107.nvidia.com (172.20.187.13) by HQMAIL109.nvidia.com (172.20.187.15) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Thu, 22 Jul 2021 04:07:07 -0700 Received: from vdi.nvidia.com (172.20.187.5) by mail.nvidia.com (172.20.187.13) with Microsoft SMTP Server id 15.0.1497.2 via Frontend Transport; Thu, 22 Jul 2021 11:07:01 +0000 From: Boris Pismenny To: , , , , , , , , , , CC: , , , , , Subject: [PATCH v5 net-next 34/36] net/mlx5e: NVMEoTCP DDGST TX handle OOO packets Date: Thu, 22 Jul 2021 14:03:23 +0300 Message-ID: <20210722110325.371-35-borisp@nvidia.com> X-Mailer: git-send-email 2.24.1 In-Reply-To: <20210722110325.371-1-borisp@nvidia.com> References: <20210722110325.371-1-borisp@nvidia.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: b12c94ab-eb95-4c2d-2f3d-08d94d00d9a2 X-MS-TrafficTypeDiagnostic: MN2PR12MB4015: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:330; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: MXotzxsRtexz+2l9FSZ/XP/tscRXCm0Q/Pmrpm1WZRNb86HrPhHgfTaDBT8KY85WGfiLUv4ebMcio7nufc0ZrZjnP/nsXBIERKg3JqHgDZzp5ERkboMkNrpFoDUeGqIU4JTRvIJRk+CpovqxFufd4ohfsPyTdhQZ+U1UL5xpsSXUqysB3rCjPrMkA7rbTjmKLn6IL5j9Q7gNtSHmS7EUpN5RQ+bHHSxzxbpWVXRahKS9tfFF84hEAZlF7sgCvARb0p6Llx6IYKVheNe7RwuKHQZ0cZTq3rkPQIhWOlr2WA+2sFPjhgg62BmtGUdssgHxO/r7qfNOTi0jrgsLHdc5+HLPHzTGwPu6RFsV9rGi2OCZ5yjs1t5Z1not6ebjTlAj19SjOxkiDgJSl7AH2KsBbC83LbJRutEADY9M2g9wBJcNlumdrS+VuAU8gnZc/goazE611W4CWyIMSyrb/gAom+4CfYNET04gbt1QOavTIb+wRaY6Qn8SxVDUHiorZvctg6Mctq3BYPeP7NlcIhPe54ZKuSQe7J9khAfjF0zQeOWjP7dFqcgTy37Lz/4RBNLKAtud7NZyu5jk4OGg2/iphJHPOHHYOiS5T9ajQMLLyjKpRO99wnJYK/ZsFr+veX/C/u3NbXb46SnfZGTJh1Z86jxaqlozm7Ss9Sb/NZv10PrG9n2BNzfFvZk+TiA6BW02h6Td0zIzbq/OI0OUjGzLWR4uXvf5qEQJ6IWZW18Cl+4= X-Forefront-Antispam-Report: CIP:216.228.112.32; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:schybrid01.nvidia.com; CAT:NONE; SFS:(4636009)(36840700001)(46966006)(8676002)(8936002)(356005)(7416002)(26005)(2906002)(110136005)(30864003)(6666004)(70206006)(70586007)(36756003)(82310400003)(921005)(83380400001)(316002)(1076003)(86362001)(54906003)(7696005)(5660300002)(107886003)(7636003)(36860700001)(4326008)(426003)(186003)(508600001)(47076005)(336012)(2616005); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Jul 2021 11:07:09.5289 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: b12c94ab-eb95-4c2d-2f3d-08d94d00d9a2 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.112.32]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DM6NAM11FT013.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR12MB4015 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Yoray Zack When the driver indicate an OOO NVMEoTCP Tx packet it starts OOO flow: 1. Get pdu_info from nvme-tcp. 2. Send indication to NIC (set psv)- NIC will rebuild the parse machine. 3. Send the data the NIC needs for computing the DDGST using DUMP wqes. Signed-off-by: Yoray Zack --- .../net/ethernet/mellanox/mlx5/core/en/txrx.h | 2 +- .../mellanox/mlx5/core/en_accel/nvmeotcp.c | 281 +++++++++++++++++- 2 files changed, 280 insertions(+), 3 deletions(-) diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h b/drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h index c7f979dfdd69..1f4beaac488a 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h +++ b/drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h @@ -142,7 +142,7 @@ struct mlx5e_tx_wqe_info { u8 num_wqebbs; u8 num_dma; u8 num_fifo_pkts; -#ifdef CONFIG_MLX5_EN_TLS +#if defined CONFIG_MLX5_EN_TLS || defined CONFIG_MLX5_EN_NVMEOTCP struct page *resync_dump_frag_page; enum mlx5e_dump_wqe_type type; #endif diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c index d9f6125f5dbc..f8cba90679ea 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c @@ -3,6 +3,7 @@ #include #include +#include #include #include "en_accel/nvmeotcp.h" #include "en_accel/nvmeotcp_utils.h" @@ -267,6 +268,18 @@ fill_nvmeotcp_progress_params(struct mlx5e_nvmeotcp_queue *queue, MLX5_SET(nvmeotcp_progress_params, ctx, offloading_state, 0); } +struct mlx5e_dump_wqe { + struct mlx5_wqe_ctrl_seg ctrl; + struct mlx5_wqe_data_seg data; +}; + +#define MLX5E_NVME_DUMP_WQEBBS\ + (DIV_ROUND_UP(sizeof(struct mlx5e_dump_wqe), MLX5_SEND_WQE_BB)) + +#define MLX5E_NVME_FETCH_DUMP_WQE(sq, pi) \ + ((struct mlx5e_dump_wqe *)\ + mlx5e_fetch_wqe(&(sq)->wq, pi, sizeof(struct mlx5e_dump_wqe))) + static void nvme_tx_fill_wi(struct mlx5e_txqsq *sq, u16 pi, u8 num_wqebbs, u32 num_bytes, struct page *page, enum mlx5e_dump_wqe_type type) @@ -276,9 +289,65 @@ static void nvme_tx_fill_wi(struct mlx5e_txqsq *sq, *wi = (struct mlx5e_tx_wqe_info) { .num_wqebbs = num_wqebbs, .num_bytes = num_bytes, + .resync_dump_frag_page = page, + .type = type, }; } +static void mlx5e_nvmeotcp_tx_post_fence_nop(struct mlx5e_txqsq *sq) +{ + struct mlx5_wq_cyc *wq = &sq->wq; + u16 pi = mlx5_wq_cyc_ctr2ix(wq, sq->pc); + + nvme_tx_fill_wi(sq, pi, 1, 0, NULL, MLX5E_DUMP_WQE_NVMEOTCP); + + mlx5e_post_nop_fence(wq, sq->sqn, &sq->pc); +} + +static int +nvmeotcp_post_resync_dump(struct mlx5e_txqsq *sq, skb_frag_t *frag, + u32 tisn, bool first, enum mlx5e_dump_wqe_type type) +{ + struct mlx5_wqe_ctrl_seg *cseg; + struct mlx5_wqe_data_seg *dseg; + struct mlx5e_dump_wqe *wqe; + dma_addr_t dma_addr; + u16 ds_cnt; + int fsz; + u16 pi; + + BUILD_BUG_ON(MLX5E_NVME_DUMP_WQEBBS != 1); + pi = mlx5_wq_cyc_ctr2ix(&sq->wq, sq->pc); + wqe = MLX5E_NVME_FETCH_DUMP_WQE(sq, pi); + + ds_cnt = sizeof(*wqe) / MLX5_SEND_WQE_DS; + + cseg = &wqe->ctrl; + dseg = &wqe->data; + + cseg->opmod_idx_opcode = cpu_to_be32((sq->pc << 8) | MLX5_OPCODE_DUMP); + cseg->qpn_ds = cpu_to_be32((sq->sqn << 8) | ds_cnt); + cseg->tis_tir_num = cpu_to_be32(tisn << 8); + cseg->fm_ce_se = first ? MLX5_FENCE_MODE_INITIATOR_SMALL : 0; + + fsz = skb_frag_size(frag); + dma_addr = skb_frag_dma_map(sq->pdev, frag, 0, fsz, + DMA_TO_DEVICE); + if (unlikely(dma_mapping_error(sq->pdev, dma_addr))) + return -ENOMEM; + + dseg->addr = cpu_to_be64(dma_addr); + dseg->lkey = sq->mkey_be; + dseg->byte_count = cpu_to_be32(fsz); + mlx5e_dma_push(sq, dma_addr, fsz, MLX5E_DMA_MAP_PAGE); + + nvme_tx_fill_wi(sq, pi, MLX5E_NVME_DUMP_WQEBBS, + fsz, skb_frag_page(frag), type); + sq->pc += MLX5E_NVME_DUMP_WQEBBS; + mlx5e_notify_hw(&sq->wq, sq->pc, sq->uar_map, &wqe->ctrl); + return 0; +} + void build_nvmeotcp_progress_params(struct mlx5e_nvmeotcp_queue *queue, struct mlx5e_set_nvmeotcp_progress_params_wqe *wqe, @@ -295,6 +364,7 @@ build_nvmeotcp_progress_params(struct mlx5e_nvmeotcp_queue *queue, MLX5_OPCODE_SET_PSV | (opc_mod << 24)); cseg->qpn_ds = cpu_to_be32((sqn << MLX5_WQE_CTRL_QPN_SHIFT) | PROGRESS_PARAMS_DS_CNT); + cseg->fm_ce_se = resync ? MLX5_FENCE_MODE_INITIATOR_SMALL : 0; fill_nvmeotcp_progress_params(queue, &wqe->params, seq, !is_rx); } @@ -1160,6 +1230,202 @@ void mlx5e_nvmeotcp_tx_post_param_wqes(struct mlx5e_txqsq *sq, struct sock *sk, mlx5e_nvmeotcp_tx_post_progress_params(ctx, sq, tcp_sk(sk)->copied_seq, false); } +enum mlx5e_nvmeotcp_resync_retval { + MLX5E_NVMEOTCP_RESYNC_DONE, + MLX5E_NVMEOTCP_RESYNC_FAIL, + MLX5E_NVMEOTCP_RESYNC_SKIP, +}; + +static +int mlx5e_nvmeotcp_resync_frag(struct mlx5e_nvmeotcp_queue *queue, + struct mlx5e_txqsq *sq, struct sk_buff *skb, + int i, skb_frag_t *frag, u32 seq) +{ + unsigned int orig_fsz, frag_offset = 0, n = 0; + enum mlx5e_dump_wqe_type type = MLX5E_DUMP_WQE_NVMEOTCP; + + orig_fsz = skb_frag_size(frag); + + do { + bool fence = !(i || frag_offset); + unsigned int fsz; + + n++; + fsz = min_t(unsigned int, sq->hw_mtu, orig_fsz - frag_offset); + skb_frag_size_set(frag, fsz); + if (nvmeotcp_post_resync_dump(sq, frag, queue->tisn, fence, type)) { + page_ref_add(compound_head(skb_frag_page(frag)), n - 1); + return -1; + } + + skb_frag_off_add(frag, fsz); + frag_offset += fsz; + } while (frag_offset < orig_fsz); + + page_ref_add(compound_head(skb_frag_page(frag)), n); + + return 0; +} + +static int mlx5e_nvmeotcp_resync_hdr(struct mlx5e_nvmeotcp_queue *queue, + struct mlx5e_txqsq *sq, u32 seq, + struct sk_buff *skb, int remaining, + struct ulp_ddp_pdu_info *pdu_info) +{ + skb_frag_t pdu_frag; + int size = min_t(int, remaining, pdu_info->hdr_len); + + __skb_frag_set_page(&pdu_frag, virt_to_page(pdu_info->hdr)); + skb_frag_off_set(&pdu_frag, offset_in_page(pdu_info->hdr)); + skb_frag_size_set(&pdu_frag, size); + + return mlx5e_nvmeotcp_resync_frag(queue, sq, skb, 0, &pdu_frag, seq); +} + +static void mlx5e_nvmeotcp_init_iter(struct iov_iter *iter, struct bio *bio) +{ + unsigned int bio_size; + struct bio_vec *vec; + int nsegs; + + vec = __bvec_iter_bvec(bio->bi_io_vec, bio->bi_iter); + nsegs = bio_segments(bio); + bio_size = bio->bi_iter.bi_size; + iov_iter_bvec(iter, 1, vec, nsegs, bio_size); + iter->iov_offset = 0; +} + +static int mlx5e_nvmeotcp_resync_data(struct mlx5e_nvmeotcp_queue *queue, + struct mlx5e_txqsq *sq, u32 seq, + struct sk_buff *skb, int remaining, + struct ulp_ddp_pdu_info *pdu_info) +{ + struct request *req = pdu_info->req; + struct bio *bio = req->bio; + struct iov_iter iter; + int data_remaining; + size_t data_sent = 0; + + mlx5e_nvmeotcp_init_iter(&iter, bio); + + data_remaining = min_t(int, remaining, pdu_info->data_len); + + while (data_remaining > 0) { + skb_frag_t frag; + size_t size = min_t(size_t, + iter.bvec->bv_len - iter.iov_offset + , data_remaining); + + __skb_frag_set_page(&frag, iter.bvec->bv_page); + skb_frag_off_set(&frag, iter.bvec->bv_offset + iter.iov_offset); + skb_frag_size_set(&frag, size); + data_remaining -= size; + + if (mlx5e_nvmeotcp_resync_frag(queue, sq, skb, 1, &frag, seq)) + goto err_out; + + if (!data_remaining) + break; + + data_sent += size; + iov_iter_advance(&iter, size); + if (!iov_iter_count(&iter) && data_sent < pdu_info->data_len) { + bio = bio->bi_next; + mlx5e_nvmeotcp_init_iter(&iter, bio); + } + } + + return 0; +err_out: + return -1; +} + +static int mlx5e_nvmeotcp_resync_crc(struct mlx5e_nvmeotcp_queue *queue, + struct mlx5e_txqsq *sq, u32 seq, + struct sk_buff *skb, int remaining, + struct ulp_ddp_pdu_info *pdu_info) +{ + skb_frag_t crc_frag; + u32 dummy_ddigest = 0; + + __skb_frag_set_page(&crc_frag, virt_to_page(&dummy_ddigest)); + skb_frag_off_set(&crc_frag, offset_in_page(&dummy_ddigest)); + skb_frag_size_set(&crc_frag, remaining); + return mlx5e_nvmeotcp_resync_frag(queue, sq, skb, 1, &crc_frag, seq); +} + +/* for a pdu info mapping [--------seq----] capsule + ******* send to HW [-------|seq *******************/ +static +bool mlx5e_nvmeotcp_resync_cap(struct mlx5e_nvmeotcp_queue *queue, + struct mlx5e_txqsq *sq, struct sk_buff *skb, + struct ulp_ddp_pdu_info *pdu_info, + u32 seq) +{ + int remaining = seq - pdu_info->start_seq; + int ret; + + ret = mlx5e_nvmeotcp_resync_hdr(queue, sq, seq, skb, remaining, + pdu_info); + if (unlikely(ret)) + goto err_out; + + remaining -= pdu_info->hdr_len; + if (remaining <= 0) + goto out; + + ret = mlx5e_nvmeotcp_resync_data(queue, sq, seq, skb, remaining, + pdu_info); + if (unlikely(ret)) + goto err_out; + + remaining -= pdu_info->data_len; + if (remaining <= 0) + goto out; + + ret = mlx5e_nvmeotcp_resync_crc(queue, sq, seq, skb, remaining, + pdu_info); + if (unlikely(ret)) + goto err_out; +out: + return true; +err_out: + return false; +} + +static enum mlx5e_nvmeotcp_resync_retval +mlx5e_nvmeotcp_handle_ooo_skb(struct mlx5e_nvmeotcp_queue *queue, + struct mlx5e_txqsq *sq, struct sk_buff *skb, + u32 seq, int datalen) +{ + struct ulp_ddp_pdu_info *pdu_info = NULL; + + /* ask for pdu_info that includes the tcp_seq */ + pdu_info = ulp_ddp_get_pdu_info(skb->sk, seq); + + if (!pdu_info) + return MLX5E_NVMEOTCP_RESYNC_SKIP; + + /*update NIC about resync - he will rebuild parse machine + *send psv with small fence + */ + mlx5e_nvmeotcp_tx_post_progress_params(queue, sq, pdu_info->start_seq, true); + + if (seq == pdu_info->start_seq || seq == pdu_info->end_seq) { + mlx5e_nvmeotcp_tx_post_fence_nop(sq); + return MLX5E_NVMEOTCP_RESYNC_DONE; + } + + /* post dump wqes - + * transfer the needed data to NIC HW using DUMP WQE with data [*,^] + * saved in pdu_info + */ + if (unlikely(!mlx5e_nvmeotcp_resync_cap(queue, sq, skb, pdu_info, seq))) + return MLX5E_NVMEOTCP_RESYNC_FAIL; + + return MLX5E_NVMEOTCP_RESYNC_DONE; +} + static inline bool mlx5e_is_sk_tx_device_offloaded(struct sock *sk) { /* Return True after smp_store_release assing in @@ -1199,8 +1465,19 @@ bool mlx5e_nvmeotcp_handle_tx_skb(struct net_device *netdev, mlx5e_nvmeotcp_tx_post_param_wqes(sq, skb->sk, ctx); seq = ntohl(tcp_hdr(skb)->seq); - if (unlikely(ctx->ulp_ddp_ctx.expected_seq != seq)) - goto err_out; + if (unlikely(ctx->ulp_ddp_ctx.expected_seq != seq)) { + enum mlx5e_nvmeotcp_resync_retval ret = + mlx5e_nvmeotcp_handle_ooo_skb(ctx, sq, skb, + seq, datalen); + switch (ret) { + case MLX5E_NVMEOTCP_RESYNC_DONE: + break; + case MLX5E_NVMEOTCP_RESYNC_SKIP: + goto out; + case MLX5E_NVMEOTCP_RESYNC_FAIL: + goto err_out; + } + } *nvmeotcp_tisn = ctx->tisn; ctx->ulp_ddp_ctx.expected_seq = seq + datalen; From patchwork Thu Jul 22 11:03:24 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Boris Pismenny X-Patchwork-Id: 485095 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-20.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER, INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2D9B9C63793 for ; Thu, 22 Jul 2021 11:07:29 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 0BBD860725 for ; Thu, 22 Jul 2021 11:07:29 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231857AbhGVK0w (ORCPT ); Thu, 22 Jul 2021 06:26:52 -0400 Received: from mail-dm6nam08on2066.outbound.protection.outlook.com ([40.107.102.66]:20160 "EHLO NAM04-DM6-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S231853AbhGVK0m (ORCPT ); Thu, 22 Jul 2021 06:26:42 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Odu3KXK+s7tNVZAcdY8JV+4fTbaC4Uc12lRetBSzxdv9VQYd3CshpmGz948XIns+tnxCOgEnd/KiHMkKuHWTL05RdCbdYozynAclnxpx7jG3c7FrJkEPYSeWRPt5ZRHMl1Nrf6SGq2us/PZB1V6A8dCrwjybXPMKwBkZXxtFzquibdhMV/389KP8Z1N24W/27a0XAqPi5DYfGU0VQgky0igPJDk5Y6BNt0NmHOJd5goUJmrLh13QPvCmyWyFJ8M1mvMYg5fd1w4PHBO0GRMBVVmSq0svvPO0iQGrHvlkL5qqnU/i5SH4v4ocVP4v/6T3zac/VrHEmKA7kn4gaQ8lkQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Bm331QKlAmKTku+aZJp2m5o27Sbebm/ySHZoVcSOFyI=; b=VAMy2KTayxaC6R7G+EW33oXvAktP0P861ZPbNleCYbc3dMZy1gcyi4GuapQYTvjINKQodwdnhcg+5/Dc8YUJp6XYe+oviLsur3tOlIvCE5/mMM2Y5wU9PrgzDv3I6nm9In88gIKUkbOF0U+942aZ/kho0sniiQwPw9KF9JQCiCocT6lqjMaMZwVxIBX7vQ6+ubhlWBMHmhxfwc04k1BH0XV87EcoNHXcDz0GNhoXdu1Vf20GsJBEgpB5NMNh3axgqE6MRiwltuzPZPHEr3ivK4dEz8LoN6rC6wvDw+52E0Ey62YxJvtF9/c0Qw/c98VNukrijjJrWcbDkioUmFfkdg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.35) smtp.rcpttodomain=fb.com smtp.mailfrom=nvidia.com; dmarc=pass (p=quarantine sp=none pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Bm331QKlAmKTku+aZJp2m5o27Sbebm/ySHZoVcSOFyI=; b=DqS6uRQgalx8192k1meAv6XfWHIUJygemhUmZwSsq3C0jCNieFOZNG7zIwb7z0KXG+H5hs22uMtc3DhRHxcxeOi2G7EUE2fKM6QyElEcew8C9jqVBpkfNl6ue5+P9mbupdOeuYE/H7TYYPJ78k/O1D4JquYiHRMqsvckVSQW+xI6S+WF2krvYB9S1LfgPFvYFHV90CIjtPcwHShk8pblYpm0ImtGt65OBuCunOZPN3lhr7iv5e0U30EZ5KTKhZQBGuJAxmy+tgPl9nLTf05V3ggi4Oph/Jv9a9Z3KTfvrdpgymbsUETemzntXCbBowsyzwvvmsWmN0LKuKU/EaEAmA== Received: from DM6PR06CA0033.namprd06.prod.outlook.com (2603:10b6:5:120::46) by BY5PR12MB4148.namprd12.prod.outlook.com (2603:10b6:a03:208::23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4331.23; Thu, 22 Jul 2021 11:07:16 +0000 Received: from DM6NAM11FT040.eop-nam11.prod.protection.outlook.com (2603:10b6:5:120:cafe::a3) by DM6PR06CA0033.outlook.office365.com (2603:10b6:5:120::46) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4352.26 via Frontend Transport; Thu, 22 Jul 2021 11:07:16 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.35) smtp.mailfrom=nvidia.com; fb.com; dkim=none (message not signed) header.d=none; fb.com; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.35 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.35; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.35) by DM6NAM11FT040.mail.protection.outlook.com (10.13.173.133) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4352.24 via Frontend Transport; Thu, 22 Jul 2021 11:07:16 +0000 Received: from HQMAIL105.nvidia.com (172.20.187.12) by HQMAIL111.nvidia.com (172.20.187.18) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Thu, 22 Jul 2021 11:07:15 +0000 Received: from HQMAIL107.nvidia.com (172.20.187.13) by HQMAIL105.nvidia.com (172.20.187.12) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Thu, 22 Jul 2021 11:07:15 +0000 Received: from vdi.nvidia.com (172.20.187.5) by mail.nvidia.com (172.20.187.13) with Microsoft SMTP Server id 15.0.1497.2 via Frontend Transport; Thu, 22 Jul 2021 11:07:08 +0000 From: Boris Pismenny To: , , , , , , , , , , CC: , , , , , Subject: [PATCH v5 net-next 35/36] net/mlx5e: NVMEoTCP DDGST TX offload optimization Date: Thu, 22 Jul 2021 14:03:24 +0300 Message-ID: <20210722110325.371-36-borisp@nvidia.com> X-Mailer: git-send-email 2.24.1 In-Reply-To: <20210722110325.371-1-borisp@nvidia.com> References: <20210722110325.371-1-borisp@nvidia.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 371a4635-8d1e-4382-9a46-08d94d00dda6 X-MS-TrafficTypeDiagnostic: BY5PR12MB4148: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:3173; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: q8cR7d2DgHGos9DHKpWzPI46BSgnQcF6FD/CjbW/VW9rsPc1NrlMyS+C7g+CxRzCKjXkGXVj6VXeCGCoiGU+1ZWVRhP0jW77sXo/vQgAyRnZiWzWqgufLoyD4Ls8U53kkTsjeJ7OMhQ9vj3D3mv0la44UOxZ9due/7VKsBhbno14GqCQgX7X+qn8ve0yadOrJnczLrdsbAhlCOMPJeXTOi7rp2LwtG/Y3cq76pJu2TNTOFAKYwD0rKXM3oC7G30bcxEG1YbooEw/8ReBATBJOqh3MBMxszxJuqbQHimFCZJzyexnQPcy2ZyluCVZtFUH6qsHu+UdM1N2EFIgQcWgw8SQIT5acDjqspBhyU5fsEWFpgMVL6+abbHGNMKg6b3HoSpNv6scqpFPtnoVnA31vonZiizUz/TOa0uXlPw/D4StYtPNnNPzo0dnMGWvHCYuFkyO/4cQX7P/hVJehEK77uwDCvZfTigzj0q7QNIdECOKx4BfZI9BZxPW5Umgo04j02qKaxBDuLfQ9kFt9xE8/YyWXPwGzWq0ypOyKAxQhoX27I2Xfp5pcl29f51NNWtQXE3cTv9OjwrkbxB+yJ0aIvSM/W20RBq49MqEnVhHNycIZ0QNGNFWveE1I3k0pDMainZJmzAIPG9GyUuWuZTBzh4Cp3itjoQ5hbO68BQBXDrnP4ipRWUdHGqIadhaMjpfWqfgGfm2u1ZoVcgG21KTQQyw7o81NwwTBw/69htyxrc= X-Forefront-Antispam-Report: CIP:216.228.112.35; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:schybrid02.nvidia.com; CAT:NONE; SFS:(4636009)(36840700001)(46966006)(36756003)(26005)(4326008)(7696005)(107886003)(2906002)(110136005)(921005)(5660300002)(36860700001)(6666004)(7636003)(8676002)(7416002)(86362001)(70206006)(336012)(82310400003)(316002)(356005)(54906003)(8936002)(2616005)(70586007)(36906005)(47076005)(426003)(186003)(83380400001)(508600001)(1076003); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Jul 2021 11:07:16.2248 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 371a4635-8d1e-4382-9a46-08d94d00dda6 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.112.35]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DM6NAM11FT040.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY5PR12MB4148 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Yoray Zack Stop offloading NVMEoTCP OOO packets which aren't contains the pdu DDGST field. When the driver indicate an OOO NVMEoTCP packet he check if it contains NVMEoTCP PDU DDGST field, if so he offloads it, otherwise he isn't. Signed-off-by: Yoray Zack --- .../mellanox/mlx5/core/en_accel/nvmeotcp.c | 23 +++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c index f8cba90679ea..f3ef92167e25 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c @@ -1393,6 +1393,20 @@ bool mlx5e_nvmeotcp_resync_cap(struct mlx5e_nvmeotcp_queue *queue, return false; } +static +bool mlx5e_nvmeotcp_check_if_need_offload(struct mlx5e_nvmeotcp_queue *queue, + u32 end_seq, u32 start_seq) +{ + if (!queue->end_seq_hint) + return false; + + /* check if skb end after pdu crc start */ + if (before(queue->end_seq_hint, end_seq) || before(start_seq, queue->start_pdu_hint)) + return false; + + return true; +} + static enum mlx5e_nvmeotcp_resync_retval mlx5e_nvmeotcp_handle_ooo_skb(struct mlx5e_nvmeotcp_queue *queue, struct mlx5e_txqsq *sq, struct sk_buff *skb, @@ -1400,12 +1414,21 @@ mlx5e_nvmeotcp_handle_ooo_skb(struct mlx5e_nvmeotcp_queue *queue, { struct ulp_ddp_pdu_info *pdu_info = NULL; + if (mlx5e_nvmeotcp_check_if_need_offload(queue, seq + datalen, seq)) { + return MLX5E_NVMEOTCP_RESYNC_SKIP; + /* ask for pdu_info that includes the tcp_seq */ pdu_info = ulp_ddp_get_pdu_info(skb->sk, seq); if (!pdu_info) return MLX5E_NVMEOTCP_RESYNC_SKIP; + queue->end_seq_hint = pdu_info->end_seq - 4; + queue->start_pdu_hint = pdu_info->start_seq; + /* check if this packet contain crc - if so offload else no */ + if (mlx5e_nvmeotcp_check_if_need_offload(queue, seq + datalen, seq)) { + return MLX5E_NVMEOTCP_RESYNC_SKIP; + /*update NIC about resync - he will rebuild parse machine *send psv with small fence */ From patchwork Thu Jul 22 11:03:25 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Boris Pismenny X-Patchwork-Id: 484308 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-20.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER, INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 600E5C6377D for ; Thu, 22 Jul 2021 11:07:32 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 38FAE60725 for ; Thu, 22 Jul 2021 11:07:31 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231866AbhGVK0y (ORCPT ); Thu, 22 Jul 2021 06:26:54 -0400 Received: from mail-bn7nam10on2074.outbound.protection.outlook.com ([40.107.92.74]:52416 "EHLO NAM10-BN7-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S231634AbhGVK0u (ORCPT ); Thu, 22 Jul 2021 06:26:50 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=A0CUYgT7Hpc6Max/+lpTDwJKAcqPHEI7Ff7XkyoB4wOivUHpFA5CKvdCx6SJUK1rhHad5vKgLUHptpCcjhOqC03xqwBqzYpKwGjxBBQaxIDAW3leNTQnpI3XNyq8PjE4gVvEINN5nQPHH1XVMiKrK2BUkYzx1O1o8JWocUI92QAkjCBVT1g8OAeP1muX9aQUmG1OBb0A1amLWWGV6mywlHULie4UJ548Km+rrnA2dc6NtJ/guHex3Ks+ksa0qyobzv9s6N53iSvft1aUSxQ4kM5LCTV0odlC7TRX40ZOIX6vGo9NJ1/FxfjL/sWtNbqX7B2Tmg9yTFpWn62W0Q28vg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=fmrIrnJiJ9al4+ThgYzDCiTuGDU5hVklX5lHdBbm4hU=; b=D8p69yeoVZYvd8deJYEQhvkTtpNbbH4Kkes8QQR62X2ZTczl2g2UnMYaBZ/0wqZ6VCtOtUO27YKVojbIBfKWgS9KifJJvIR47ZYgsTan+77wnqMqrCmHNpWhaKwCOfYy7qq5JnGy0US1H+soEIKs+t/lRV0I/LBCxL2/pmHtI2LoJkoNLC/CjI8Us48i220JBRGJNHDK2zhB5WdcGufswSYsPWXEFBB4/M4x4z02uHxejaEhpAIDiS0wf4kjB7JLPCm11Xj62WlJm3na48wRX5WdM6DJ68u6qVGrsKJp+uv85Gdy+XvNHPb9hIptuRmzR4hjwze3RT6yGGPfPHzueQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.32) smtp.rcpttodomain=marvell.com smtp.mailfrom=nvidia.com; dmarc=pass (p=quarantine sp=none pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=fmrIrnJiJ9al4+ThgYzDCiTuGDU5hVklX5lHdBbm4hU=; b=b8bXfZdaL6QFMCRW2YYmtK1U0b7qC/ISsIYH5eNPqfwHzAxGsyzfaP/CxG0KauTwJ8D/4XP6C1JzLydjy7ZpDIGL82tX8IhPe0OZvJxVu5hTA3VIeFCdw2oFg2dsWikLqJcai9Ij0cjnxvBHVyw4UraTG4pR4hf7FHhGxsSHfICEQ6RAXFExDMSUObPFwEEFZAAOtxzT7+3spAgKvSjOWHCasXiClc5BKq+7Qzzo3zC4JcWWQrxUZPz0o2PIB1Dys1qrAHSoEI4hE77udciEUgxbptUtf5WlRb/SLntXfQPnAgcCsQU9a8vDFR5d9tOSy11NUI1uKCrfClieJGO3ig== Received: from DM6PR02CA0075.namprd02.prod.outlook.com (2603:10b6:5:1f4::16) by BN9PR12MB5163.namprd12.prod.outlook.com (2603:10b6:408:11c::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4352.26; Thu, 22 Jul 2021 11:07:24 +0000 Received: from DM6NAM11FT022.eop-nam11.prod.protection.outlook.com (2603:10b6:5:1f4:cafe::3e) by DM6PR02CA0075.outlook.office365.com (2603:10b6:5:1f4::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4352.25 via Frontend Transport; Thu, 22 Jul 2021 11:07:24 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.32) smtp.mailfrom=nvidia.com; marvell.com; dkim=none (message not signed) header.d=none; marvell.com; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.32 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.32; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.32) by DM6NAM11FT022.mail.protection.outlook.com (10.13.172.210) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4352.24 via Frontend Transport; Thu, 22 Jul 2021 11:07:23 +0000 Received: from HQMAIL107.nvidia.com (172.20.187.13) by HQMAIL109.nvidia.com (172.20.187.15) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Thu, 22 Jul 2021 04:07:22 -0700 Received: from vdi.nvidia.com (172.20.187.5) by mail.nvidia.com (172.20.187.13) with Microsoft SMTP Server id 15.0.1497.2 via Frontend Transport; Thu, 22 Jul 2021 11:07:15 +0000 From: Boris Pismenny To: , , , , , , , , , , CC: , , , , , Subject: [PATCH v5 net-next 36/36] net/mlx5e: NVMEoTCP DDGST TX statistics Date: Thu, 22 Jul 2021 14:03:25 +0300 Message-ID: <20210722110325.371-37-borisp@nvidia.com> X-Mailer: git-send-email 2.24.1 In-Reply-To: <20210722110325.371-1-borisp@nvidia.com> References: <20210722110325.371-1-borisp@nvidia.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 582051a0-d58a-412b-5d31-08d94d00e20f X-MS-TrafficTypeDiagnostic: BN9PR12MB5163: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:390; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: fNZ/hH39b2ZNDuxq01a4fB7g64ju9ssYzLimfx7NFDpK3hIq7R0flgWB1DLcHvkAwok6z+WY+D0x+STXg9pRc6VQdpRTpArsNRmAxJqLrHN29GIsp/OZNqWTy65936wzJijX2HR9Dh7HJy5uaULlckrDIufJhD+wxFTNxKd725ydgNy4P2Zs8ne2QCFs3CkDKLSxX2J2KNqfbrM99ELBVYzx6DzSTsgfm6vQdjWnJTET94Hb4ZRM3SPWGZpF4eNZkbKRT+R1k7MrkOyoO6soPC8fCvHaV2HwbMMQZeeTwaZvgNnZpE2WHGrFyR3XQi06Dub12OiP8LOFGsVpMmK1SvX9aJ6CYxvMz1lgFmK54n62g9PrRpBIeV2k6L+hsmVK4hhNzu8BBnnZNNdtVjGro5H5HEbm1//ykSZ6kFzj9Wle75okPbHS6Maz+3tUBjDKRJZwC0Hu3cKJ48u3SXDWXtzJ5roAPrel61oNfte5segMqS1DJoC6AmwrYtWZcg7bmE6PoAPh+kWt6SjX8RbzBl+voP8IH0diz61Y9vRaJbVoUX/PlqQKVEPwO4Z63HdbInm2UDUUY8NCpWBtMbTvOKoc27Lk4f8gIwp5y20ylPPx31mi7f71icLAUPKiRBbAlpLRacA19rfyPI+A2mj/S3KK4+LvJvocG22Nn766783w74C+GcLrTTH1xmz/jtwakIBM2W37Sk5OK0eoFH58ED1ZCx7P1y9vPJ0tstuSOxY= X-Forefront-Antispam-Report: CIP:216.228.112.32; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:schybrid01.nvidia.com; CAT:NONE; SFS:(4636009)(36840700001)(46966006)(356005)(6666004)(47076005)(83380400001)(110136005)(336012)(36860700001)(186003)(7636003)(26005)(82310400003)(54906003)(86362001)(921005)(7696005)(2906002)(7416002)(8676002)(1076003)(4326008)(8936002)(5660300002)(426003)(2616005)(316002)(508600001)(70586007)(70206006)(107886003)(36756003); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Jul 2021 11:07:23.6123 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 582051a0-d58a-412b-5d31-08d94d00e20f X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.112.32]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DM6NAM11FT022.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN9PR12MB5163 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Yoray Zack NVMEoTCP Tx offload statistics includes both control and data path statistic: counters for contex, offloaded packets/bytes, out-of-order packets resync operation (success/fail), and DUMP packets/bytes. Signed-off-by: Yoray Zack --- .../mellanox/mlx5/core/en_accel/ktls_tx.c | 4 ++ .../mellanox/mlx5/core/en_accel/nvmeotcp.c | 22 ++++++++++- .../ethernet/mellanox/mlx5/core/en_stats.c | 37 +++++++++++++++++++ .../ethernet/mellanox/mlx5/core/en_stats.h | 23 ++++++++++++ 4 files changed, 84 insertions(+), 2 deletions(-) diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_tx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_tx.c index 64780d0143ec..4b0d4bd88b9c 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_tx.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_tx.c @@ -366,6 +366,10 @@ void mlx5e_ktls_tx_handle_resync_dump_comp(struct mlx5e_txqsq *sq, stats->tls_dump_bytes += wi->num_bytes; break; case MLX5E_DUMP_WQE_NVMEOTCP: +#ifdef CONFIG_ULP_DDP + stats->nvmeotcp_dump_packets++; + stats->nvmeotcp_dump_bytes += wi->num_bytes; +#endif break; } } diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c index f3ef92167e25..34676c81d889 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c @@ -1388,8 +1388,10 @@ bool mlx5e_nvmeotcp_resync_cap(struct mlx5e_nvmeotcp_queue *queue, if (unlikely(ret)) goto err_out; out: + sq->stats->nvmeotcp_resync++; return true; err_out: + sq->stats->nvmeotcp_resync_fail++; return false; } @@ -1413,21 +1415,29 @@ mlx5e_nvmeotcp_handle_ooo_skb(struct mlx5e_nvmeotcp_queue *queue, u32 seq, int datalen) { struct ulp_ddp_pdu_info *pdu_info = NULL; + struct mlx5e_sq_stats *stats = sq->stats; + stats->nvmeotcp_ooo++; if (mlx5e_nvmeotcp_check_if_need_offload(queue, seq + datalen, seq)) { + stats->nvmeotcp_no_need_offload++; return MLX5E_NVMEOTCP_RESYNC_SKIP; + } /* ask for pdu_info that includes the tcp_seq */ pdu_info = ulp_ddp_get_pdu_info(skb->sk, seq); - if (!pdu_info) + if (!pdu_info) { + stats->nvmeotcp_no_pdu_info++; return MLX5E_NVMEOTCP_RESYNC_SKIP; + } queue->end_seq_hint = pdu_info->end_seq - 4; queue->start_pdu_hint = pdu_info->start_seq; /* check if this packet contain crc - if so offload else no */ if (mlx5e_nvmeotcp_check_if_need_offload(queue, seq + datalen, seq)) { + stats->nvmeotcp_no_need_offload++; return MLX5E_NVMEOTCP_RESYNC_SKIP; + } /*update NIC about resync - he will rebuild parse machine *send psv with small fence @@ -1464,6 +1474,7 @@ bool mlx5e_nvmeotcp_handle_tx_skb(struct net_device *netdev, struct sk_buff *skb, int *nvmeotcp_tisn) { struct mlx5e_nvmeotcp_queue *ctx; + struct mlx5e_sq_stats *stats = sq->stats; int datalen; u32 seq; @@ -1484,8 +1495,10 @@ bool mlx5e_nvmeotcp_handle_tx_skb(struct net_device *netdev, if (WARN_ON_ONCE(ctx->ulp_ddp_ctx.netdev != netdev)) goto err_out; - if (unlikely(mlx5e_nvmeotcp_test_and_clear_pending(ctx))) + if (unlikely(mlx5e_nvmeotcp_test_and_clear_pending(ctx))) { mlx5e_nvmeotcp_tx_post_param_wqes(sq, skb->sk, ctx); + stats->nvmeotcp_ctx++; + } seq = ntohl(tcp_hdr(skb)->seq); if (unlikely(ctx->ulp_ddp_ctx.expected_seq != seq)) { @@ -1504,6 +1517,11 @@ bool mlx5e_nvmeotcp_handle_tx_skb(struct net_device *netdev, *nvmeotcp_tisn = ctx->tisn; ctx->ulp_ddp_ctx.expected_seq = seq + datalen; + stats->nvmeotcp_offload_packets += skb_is_gso(skb) ? + skb_shinfo(skb)->gso_segs : 1; + + stats->nvmeotcp_offload_bytes += datalen; + goto good_out; out: *nvmeotcp_tisn = 0; diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_stats.c b/drivers/net/ethernet/mellanox/mlx5/core/en_stats.c index 3d95e46422e5..c51d1a76b22f 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_stats.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_stats.c @@ -125,6 +125,18 @@ static const struct counter_desc sw_stats_desc[] = { { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, tx_tls_drop_no_sync_data) }, { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, tx_tls_drop_bypass_req) }, #endif +#ifdef CONFIG_MLX5_EN_NVMEOTCP + { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, tx_nvmeotcp_offload_packets) }, + { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, tx_nvmeotcp_offload_bytes) }, + { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, tx_nvmeotcp_ooo) }, + { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, tx_nvmeotcp_dump_packets) }, + { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, tx_nvmeotcp_dump_bytes) }, + { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, tx_nvmeotcp_resync) }, + { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, tx_nvmeotcp_ctx) }, + { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, tx_nvmeotcp_resync_fail) }, + { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, tx_nvmeotcp_no_need_offload) }, + { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, tx_nvmeotcp_no_pdu_info) }, +#endif { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_lro_packets) }, { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_lro_bytes) }, @@ -420,6 +432,19 @@ static void mlx5e_stats_grp_sw_update_stats_sq(struct mlx5e_sw_stats *s, s->tx_tls_drop_no_sync_data += sq_stats->tls_drop_no_sync_data; s->tx_tls_drop_bypass_req += sq_stats->tls_drop_bypass_req; #endif +#ifdef CONFIG_MLX5_EN_NVMEOTCP + s->tx_nvmeotcp_offload_packets += sq_stats->nvmeotcp_offload_packets; + s->tx_nvmeotcp_offload_bytes += sq_stats->nvmeotcp_offload_bytes; + s->tx_nvmeotcp_ooo += sq_stats->nvmeotcp_ooo; + s->tx_nvmeotcp_dump_bytes += sq_stats->nvmeotcp_dump_bytes; + s->tx_nvmeotcp_dump_packets += sq_stats->nvmeotcp_dump_packets; + s->tx_nvmeotcp_resync += sq_stats->nvmeotcp_resync; + s->tx_nvmeotcp_ctx += sq_stats->nvmeotcp_ctx; + s->tx_nvmeotcp_resync_fail += sq_stats->nvmeotcp_resync_fail; + s->tx_nvmeotcp_no_need_offload += sq_stats->nvmeotcp_no_need_offload; + s->tx_nvmeotcp_no_pdu_info += sq_stats->nvmeotcp_no_pdu_info; +#endif + s->tx_cqes += sq_stats->cqes; } @@ -1850,6 +1875,18 @@ static const struct counter_desc sq_stats_desc[] = { { MLX5E_DECLARE_TX_STAT(struct mlx5e_sq_stats, tls_skip_no_sync_data) }, { MLX5E_DECLARE_TX_STAT(struct mlx5e_sq_stats, tls_drop_no_sync_data) }, { MLX5E_DECLARE_TX_STAT(struct mlx5e_sq_stats, tls_drop_bypass_req) }, +#endif +#ifdef CONFIG_MLX5_EN_NVMEOTCP + { MLX5E_DECLARE_TX_STAT(struct mlx5e_sq_stats, nvmeotcp_offload_packets) }, + { MLX5E_DECLARE_TX_STAT(struct mlx5e_sq_stats, nvmeotcp_offload_bytes) }, + { MLX5E_DECLARE_TX_STAT(struct mlx5e_sq_stats, nvmeotcp_ooo) }, + { MLX5E_DECLARE_TX_STAT(struct mlx5e_sq_stats, nvmeotcp_dump_packets) }, + { MLX5E_DECLARE_TX_STAT(struct mlx5e_sq_stats, nvmeotcp_dump_bytes) }, + { MLX5E_DECLARE_TX_STAT(struct mlx5e_sq_stats, nvmeotcp_resync) }, + { MLX5E_DECLARE_TX_STAT(struct mlx5e_sq_stats, nvmeotcp_ctx) }, + { MLX5E_DECLARE_TX_STAT(struct mlx5e_sq_stats, nvmeotcp_no_need_offload) }, + { MLX5E_DECLARE_TX_STAT(struct mlx5e_sq_stats, nvmeotcp_no_pdu_info) }, + { MLX5E_DECLARE_TX_STAT(struct mlx5e_sq_stats, nvmeotcp_resync_fail) }, #endif { MLX5E_DECLARE_TX_STAT(struct mlx5e_sq_stats, csum_none) }, { MLX5E_DECLARE_TX_STAT(struct mlx5e_sq_stats, stopped) }, diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_stats.h b/drivers/net/ethernet/mellanox/mlx5/core/en_stats.h index e645ee83de97..3ca48d69a2d6 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_stats.h +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_stats.h @@ -205,6 +205,17 @@ struct mlx5e_sw_stats { u64 rx_nvmeotcp_resync; u64 rx_nvmeotcp_offload_packets; u64 rx_nvmeotcp_offload_bytes; + + u64 tx_nvmeotcp_offload_packets; + u64 tx_nvmeotcp_offload_bytes; + u64 tx_nvmeotcp_ooo; + u64 tx_nvmeotcp_resync; + u64 tx_nvmeotcp_dump_packets; + u64 tx_nvmeotcp_dump_bytes; + u64 tx_nvmeotcp_ctx; + u64 tx_nvmeotcp_no_need_offload; + u64 tx_nvmeotcp_no_pdu_info; + u64 tx_nvmeotcp_resync_fail; #endif u64 ch_events; u64 ch_poll; @@ -405,6 +416,18 @@ struct mlx5e_sq_stats { u64 tls_skip_no_sync_data; u64 tls_drop_no_sync_data; u64 tls_drop_bypass_req; +#endif +#ifdef CONFIG_MLX5_EN_NVMEOTCP + u64 nvmeotcp_offload_packets; + u64 nvmeotcp_offload_bytes; + u64 nvmeotcp_ooo; + u64 nvmeotcp_resync; + u64 nvmeotcp_dump_packets; + u64 nvmeotcp_dump_bytes; + u64 nvmeotcp_ctx; + u64 nvmeotcp_resync_fail; + u64 nvmeotcp_no_need_offload; + u64 nvmeotcp_no_pdu_info; #endif /* less likely accessed in data path */ u64 csum_none;