From patchwork Thu Apr 8 09:10:43 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eli Cohen X-Patchwork-Id: 418721 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER, INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9E558C43462 for ; Thu, 8 Apr 2021 09:11:02 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 44EC761107 for ; Thu, 8 Apr 2021 09:11:02 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230342AbhDHJLK (ORCPT ); Thu, 8 Apr 2021 05:11:10 -0400 Received: from mail-eopbgr770087.outbound.protection.outlook.com ([40.107.77.87]:25454 "EHLO NAM02-SN1-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S229603AbhDHJLI (ORCPT ); Thu, 8 Apr 2021 05:11:08 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=YNe66yfVnG1GwoLdU4X8lfuKZcmw0/Itjwrilc/lPo8ZQLeQM14Za7Uuh4v5C9/c4c9Mq0V+62gBwSgh/QwXu/LsNlnFajYF4xH/ui005BUVi5HrlA300SYsKPXpWWJ0gf4GMytYOmFyro8zBSwdxMTsRchcu/iDZmk47g56+VxhY4BcXgK2IikTOjTJ0yW3tq00Wfd9WgDERMctiwYvdx5hTzAgGo1rlPChrqizuDlIecRUAcQpZ/U90So/LaEL8FVWihAHPfkkQktfzfiQ1/nI24/XEpKud4AxW6ovFuPqqPCj9pmfabu5ic/kvjJUwI8W3/fx+zlFmjLwSfQnxw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=DzZEAR5poXPGX/1jigS6ffEhCHKkS16g+x/r0a3YBx4=; b=PyWRe8LJ3z8/vneQWP5AEooIn0kH+h/7u+VxpQgsAN7Tlv1x1b3HES842Q4OetKSEHVUhny+8Y3z4o89VPL/yOImYNxyX+GuS3McRG0B9idTdDvcP6osMYywzgpfIMJ36mdR9+W1Aa2i3t5H1uxu+cPJbZ7gyif1DOGtf63Lsgai7qaG0moFyRx5NCsL8qSNvYv0WiiaoH2FPxqrSc50jqkCqhpMHqtm0X9wkjFLsZwP3OTxWVgbqTqgHXvyZHybqCiXZhsRg1EQCFEL33QbPuIVrwlt+SYboIT9Wigy4xAKhCq3IniYT7qwmCq711n0lrQczOfXNy7q3avMuKmj+w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.36) smtp.rcpttodomain=lists.linux-foundation.org smtp.mailfrom=nvidia.com; dmarc=pass (p=none sp=none pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=DzZEAR5poXPGX/1jigS6ffEhCHKkS16g+x/r0a3YBx4=; b=Mipk4fhCm/QrupRHmSL+L2kl8UtHwdvsf3oX6y3ozbR5EuoiGeigPjQPD56J1OH3Mhg/TFPDPuQMv6R2rHSdvXkYv7GeJooO82fIc7U07PTbo0F9c72XP3sNTYqNVTafFlfqk9UanPa6hG9/AKQrtJJ0tKofnaS/ds9U8whi7uZoEj0UTjClrXYbYf+EFfBAOPlzIPF8JVvzl4W3cFtR2FTBgpZTQwZegkDrsTaYc71Z4rZI5HTSCJWcGe5i3IKLjtMsa9httzgnbgd26+gdp8E/o3dl/0U2nHYkojPOFnx1WZ4Cwp8icvunUIiH6+BnBR+miBjpzIOtwSRJspPDKw== Received: from MW4PR03CA0306.namprd03.prod.outlook.com (2603:10b6:303:dd::11) by PH0PR12MB5404.namprd12.prod.outlook.com (2603:10b6:510:d7::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4020.16; Thu, 8 Apr 2021 09:10:56 +0000 Received: from CO1NAM11FT065.eop-nam11.prod.protection.outlook.com (2603:10b6:303:dd:cafe::e) by MW4PR03CA0306.outlook.office365.com (2603:10b6:303:dd::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4020.16 via Frontend Transport; Thu, 8 Apr 2021 09:10:56 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.36) smtp.mailfrom=nvidia.com; lists.linux-foundation.org; dkim=none (message not signed) header.d=none; lists.linux-foundation.org; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.36 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.36; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.36) by CO1NAM11FT065.mail.protection.outlook.com (10.13.174.62) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4020.17 via Frontend Transport; Thu, 8 Apr 2021 09:10:56 +0000 Received: from HQMAIL105.nvidia.com (172.20.187.12) by HQMAIL101.nvidia.com (172.20.187.10) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Thu, 8 Apr 2021 09:10:55 +0000 Received: from vdi.nvidia.com (172.20.145.6) by mail.nvidia.com (172.20.187.12) with Microsoft SMTP Server id 15.0.1497.2 via Frontend Transport; Thu, 8 Apr 2021 09:10:53 +0000 From: Eli Cohen To: , , , , , CC: , Eli Cohen Subject: [PATCH 1/5] vdpa/mlx5: should exclude header length and fcs from mtu Date: Thu, 8 Apr 2021 12:10:43 +0300 Message-ID: <20210408091047.4269-2-elic@nvidia.com> X-Mailer: git-send-email 2.30.1 In-Reply-To: <20210408091047.4269-1-elic@nvidia.com> References: <20210408091047.4269-1-elic@nvidia.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: df021eac-e885-4a5f-1d8d-08d8fa6e37e4 X-MS-TrafficTypeDiagnostic: PH0PR12MB5404: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:7691; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: sqtv2YOyQNBRxWxTOT1B7nOC3wlq/5oB+pahhT1x4tdXXw5nqBRCvRcUtMTXrOYhudpTDjV5Hf4CRoQkxkXhbJ5ILf5ABhMWMVKAo+pnz5Ytjj9iTYmJ6eTGwrU2g9VV3YH6fsSS6mnDnkPTCMVPLMtIjA8FBgp7EYxQ/xadR5A1SzehwepZm6Lto6s06M9rXCSmq0Xx+Es0X85/W50oAmc/xQZokPGyPSc8XVCclWK6Fn7fjyoepl/+hBFannx047V+9Juzraj67JudBqxTYS0SDKZhKih9W86j9/GxuR7SeTS5hISksy1cbas/IHWWXhvTUqs5i/K50DXNk3L6XF1UJ18zMP3zVRfuiU2muxw5L01f79jNTqYk+S78XLn/KD2G+MooNE6mEe3XxK7qPLXuOTbH6YtKwIN8cMNXU3KSxQEk1qleSf8jC7miuxtIN1KEK5PWWMMOZ0+gRN1sdCMssn+YO7r1AAwIDw2Ny62KZNhCJWrKSPGZojvK/wcSWmIAR0Ioo10DUfd8ErA+Y1djbv+NJoV/yWibG5Fzf+pqx2BWPswwSQiuVLWTilNlMHwdgT346itOIxKv6+QdNwLCQklvLLneLkjT4fY5q71Y+ugmSk7ry66J46urEaGOA5AUQOLc1HqfybgazWoOoHjsZ0fSwRDofZsZU4IyKA0= X-Forefront-Antispam-Report: CIP:216.228.112.36; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:schybrid05.nvidia.com; CAT:NONE; SFS:(4636009)(39860400002)(396003)(346002)(376002)(136003)(46966006)(36840700001)(2906002)(8936002)(336012)(86362001)(426003)(107886003)(7696005)(4326008)(1076003)(2616005)(82310400003)(186003)(36756003)(82740400003)(26005)(6666004)(356005)(70586007)(7636003)(8676002)(316002)(54906003)(36906005)(5660300002)(47076005)(83380400001)(70206006)(110136005)(36860700001)(478600001); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 08 Apr 2021 09:10:56.1449 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: df021eac-e885-4a5f-1d8d-08d8fa6e37e4 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.112.36]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CO1NAM11FT065.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH0PR12MB5404 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Si-Wei Liu When feature VIRTIO_NET_F_MTU is negotiated on mlx5_vdpa, 22 extra bytes worth of MTU length is shown in guest. This is because the mlx5_query_port_max_mtu API returns the "hardware" MTU value, which does not just contain the Ethernet payload, but includes extra lengths starting from the Ethernet header up to the FCS altogether. Fix the MTU so packets won't get dropped silently. Fixes: 1a86b377aa21 ("vdpa/mlx5: Add VDPA driver for supported mlx5 devices") Signed-off-by: Si-Wei Liu Acked-by: Jason Wang Acked-by: Eli Cohen --- drivers/vdpa/mlx5/core/mlx5_vdpa.h | 4 ++++ drivers/vdpa/mlx5/net/mlx5_vnet.c | 15 ++++++++++++++- 2 files changed, 18 insertions(+), 1 deletion(-) diff --git a/drivers/vdpa/mlx5/core/mlx5_vdpa.h b/drivers/vdpa/mlx5/core/mlx5_vdpa.h index 08f742fd2409..b6cc53ba980c 100644 --- a/drivers/vdpa/mlx5/core/mlx5_vdpa.h +++ b/drivers/vdpa/mlx5/core/mlx5_vdpa.h @@ -4,9 +4,13 @@ #ifndef __MLX5_VDPA_H__ #define __MLX5_VDPA_H__ +#include +#include #include #include +#define MLX5V_ETH_HARD_MTU (ETH_HLEN + VLAN_HLEN + ETH_FCS_LEN) + struct mlx5_vdpa_direct_mr { u64 start; u64 end; diff --git a/drivers/vdpa/mlx5/net/mlx5_vnet.c b/drivers/vdpa/mlx5/net/mlx5_vnet.c index 71397fdafa6a..a49ebb250253 100644 --- a/drivers/vdpa/mlx5/net/mlx5_vnet.c +++ b/drivers/vdpa/mlx5/net/mlx5_vnet.c @@ -1907,6 +1907,19 @@ static const struct vdpa_config_ops mlx5_vdpa_ops = { .free = mlx5_vdpa_free, }; +static int query_mtu(struct mlx5_core_dev *mdev, u16 *mtu) +{ + u16 hw_mtu; + int err; + + err = mlx5_query_nic_vport_mtu(mdev, &hw_mtu); + if (err) + return err; + + *mtu = hw_mtu - MLX5V_ETH_HARD_MTU; + return 0; +} + static int alloc_resources(struct mlx5_vdpa_net *ndev) { struct mlx5_vdpa_net_resources *res = &ndev->res; @@ -1992,7 +2005,7 @@ static int mlx5v_probe(struct auxiliary_device *adev, init_mvqs(ndev); mutex_init(&ndev->reslock); config = &ndev->config; - err = mlx5_query_nic_vport_mtu(mdev, &ndev->mtu); + err = query_mtu(mdev, &ndev->mtu); if (err) goto err_mtu; From patchwork Thu Apr 8 09:10:44 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eli Cohen X-Patchwork-Id: 418030 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER, INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B337CC4360C for ; Thu, 8 Apr 2021 09:11:07 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 6864761139 for ; Thu, 8 Apr 2021 09:11:07 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230369AbhDHJLQ (ORCPT ); Thu, 8 Apr 2021 05:11:16 -0400 Received: from mail-dm6nam10on2041.outbound.protection.outlook.com ([40.107.93.41]:1453 "EHLO NAM10-DM6-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S230360AbhDHJLM (ORCPT ); Thu, 8 Apr 2021 05:11:12 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=AA4j+xJojwzP+17Kraza7U3Wwsqe0uid5d17uH7fyHFxonNABQMcvS7GtZCeegO7jxRIT36Z9FijFNT7VtRLxWy5rZ6k9fZGdO5CVXPu4064SuFF8B582YlfviewPeEGxjRrQh6U6KIirDF5MZUuaIbyKROP7++c8MbWCtkMCOIhoNNukJfbXX6sW7wtFYUnK5HsMPdBThjxgR6cEq0CSrdpA6YnGtztYtKJUDDtQx1ZP8cxqRUxufjMJKpAbvIWvuk9uPcZ/rV6wwNhb83UvEL1ETfqtBFXOHuSF04PhLdL4bd0UFdyBVscdWP8LTqKnsZkmiuR5SQ22CAbHzik6Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=KlcZgYgrgCV/2AMLFsa9OHbkLO0EqJZiTcUFRFtyvdg=; b=m1kDnUUPJJ+5AZ1ZlkyaOYXH3mCrdrBXBBhgnbhzeGL0SP7Hq/4oz9+pKHUqjv+fg9TOZTbPW1AmwANp3vgU1sjMNIh5jzBsllOoIGw3YlBulTpEyS9RZF2N9g036lNf3LJtEi0IMvrj2R6hfc4wwwCnc7WELCtj72Q/KXiiK/YiOrkr/w0NjFhozl0FOOfqAUEu6cRlOqXAirv1MFGfNBfWFYdeIGfFYJ5ZBwfvPnNpJF8S+lDXoy3D9Zcg2FOnm01Q9iX4j67NQqox431Xo0d9K3XDzeWZU1ubEK4mq4IYnhVKM32de4WbwOa5pkamuZcmWMJP+AN/uHCR8QSwJw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.34) smtp.rcpttodomain=lists.linux-foundation.org smtp.mailfrom=nvidia.com; dmarc=pass (p=none sp=none pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=KlcZgYgrgCV/2AMLFsa9OHbkLO0EqJZiTcUFRFtyvdg=; b=Q4zbRCeiI4hLh051cwjq+Ub2mMgosx0QzA1oM29dKNgkl/KvWduLrtPQbCXE4+NA73961wTzZEFsNBo+vpLcjiKFZf9vunH5H17vhsjNxFe1KQyLJR6BInnPwyh0IosKLOEc5HHzeodtom9+Vfu0VXAgVDYRkPm48Jl+9RGHPke3jwrKj2ySqUsSwIBIf3ixSzDahdIRn2yxCxyAIgWijWHXf+uCD+gGweu6E6V+uduHcGGVq1V8bUf+bmUfgocgd9MAaMF6vgnVvnHmYqBipu7fkldDFSMqwmIJPNolfh4CM/s0+TNJk+BKor28WM1cK++q2Z24UTxfVld2RhX3Lg== Received: from BN6PR11CA0008.namprd11.prod.outlook.com (2603:10b6:405:2::18) by BYAPR12MB4709.namprd12.prod.outlook.com (2603:10b6:a03:98::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3999.26; Thu, 8 Apr 2021 09:10:59 +0000 Received: from BN8NAM11FT011.eop-nam11.prod.protection.outlook.com (2603:10b6:405:2:cafe::47) by BN6PR11CA0008.outlook.office365.com (2603:10b6:405:2::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4020.16 via Frontend Transport; Thu, 8 Apr 2021 09:10:59 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.34) smtp.mailfrom=nvidia.com; lists.linux-foundation.org; dkim=none (message not signed) header.d=none; lists.linux-foundation.org; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.34 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.34; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.34) by BN8NAM11FT011.mail.protection.outlook.com (10.13.176.140) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4020.17 via Frontend Transport; Thu, 8 Apr 2021 09:10:58 +0000 Received: from HQMAIL105.nvidia.com (172.20.187.12) by HQMAIL107.nvidia.com (172.20.187.13) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Thu, 8 Apr 2021 09:10:58 +0000 Received: from vdi.nvidia.com (172.20.145.6) by mail.nvidia.com (172.20.187.12) with Microsoft SMTP Server id 15.0.1497.2 via Frontend Transport; Thu, 8 Apr 2021 09:10:55 +0000 From: Eli Cohen To: , , , , , CC: , Eli Cohen Subject: [PATCH 2/5] vdpa/mlx5: Use the correct dma device when registering memory Date: Thu, 8 Apr 2021 12:10:44 +0300 Message-ID: <20210408091047.4269-3-elic@nvidia.com> X-Mailer: git-send-email 2.30.1 In-Reply-To: <20210408091047.4269-1-elic@nvidia.com> References: <20210408091047.4269-1-elic@nvidia.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 14dfed17-a01f-45c6-c2bc-08d8fa6e397a X-MS-TrafficTypeDiagnostic: BYAPR12MB4709: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:565; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 96FszvgrHwYyygsWfNLZotPmTOdaH8zB4bx7mYjq0Cs0bbbbUfZ9b53DiuV/bdFJrYOPt9R7uaagzgjWLWPe3kJLUobWDbZg1PfEA3G0Z8EKULydVArlHzfP3vKaWo9GM/VG+Ch8Pt0J/rtiLox/C3VDRx24UsPn3wxYVvGYYexC/SKYjGy9nVpAd9MtKrRHBeAB6XBH8rKF4zx/XdvJXH3C/WvNOAVTHu3KdPTcVtTiRgPtqxFcJgPSrJHizWrdkvyJSxlGb9TrbPzPWaCfCvHcKhNQUVlA+Mnx4iItGxsSMcjWZ8FArNjx6CfcWdItqxNViKFmUqkteAbmbS0Cmm4P1jxkEcaaqGUi08nHxutLeRpwE0ORRzojUpSeJ18i6I8ZJrcAjczPLXMF3+qX9OMmOmjY8RZMoaQTV7WTOqBOuWh0yWSHsIXk/0125Sb9/ErZOgpLzUP6HCbv63RpxJXwYSU2+4S/QnqSonFhYbwmV57shWWvkY98ykGFcN+pnxzq6ntxoMr73h83yJa51GM3Y/TQYFvIqi9hKA8RNkoJQsURRTgsFx2ZpODzB5QYcg7d8Bjv+DTenzQ/pootA9b4DXJ6uVqZucAdCu1eHlF+vz2yOz/M7JGD72IbaQY68znjHNcIdHx0mfxZFbnTGIbLObkSMEZk2Z0h7D/T+mQ= X-Forefront-Antispam-Report: CIP:216.228.112.34; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:schybrid03.nvidia.com; CAT:NONE; SFS:(4636009)(346002)(39860400002)(376002)(136003)(396003)(36840700001)(46966006)(356005)(107886003)(82310400003)(110136005)(36906005)(26005)(1076003)(336012)(2906002)(54906003)(7636003)(316002)(70586007)(4326008)(6666004)(70206006)(47076005)(86362001)(36860700001)(7696005)(8676002)(186003)(2616005)(8936002)(36756003)(426003)(83380400001)(82740400003)(478600001)(5660300002); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 08 Apr 2021 09:10:58.8829 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 14dfed17-a01f-45c6-c2bc-08d8fa6e397a X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.112.34]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BN8NAM11FT011.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR12MB4709 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org In cases where the vdpa instance uses a SF (sub function), the DMA device is the parent device. Use a function to retrieve the correct DMA device. Fixes: 1958fc2f0712 ("net/mlx5: SF, Add auxiliary device driver") Signed-off-by: Eli Cohen Reviewed-by: Parav Pandit --- drivers/vdpa/mlx5/core/mr.c | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/drivers/vdpa/mlx5/core/mr.c b/drivers/vdpa/mlx5/core/mr.c index d300f799efcd..3908ff28eec0 100644 --- a/drivers/vdpa/mlx5/core/mr.c +++ b/drivers/vdpa/mlx5/core/mr.c @@ -219,6 +219,11 @@ static void destroy_indirect_key(struct mlx5_vdpa_dev *mvdev, struct mlx5_vdpa_m mlx5_vdpa_destroy_mkey(mvdev, &mkey->mkey); } +static struct device *get_dma_device(struct mlx5_vdpa_dev *mvdev) +{ + return &mvdev->mdev->pdev->dev; +} + static int map_direct_mr(struct mlx5_vdpa_dev *mvdev, struct mlx5_vdpa_direct_mr *mr, struct vhost_iotlb *iotlb) { @@ -234,7 +239,7 @@ static int map_direct_mr(struct mlx5_vdpa_dev *mvdev, struct mlx5_vdpa_direct_mr u64 pa; u64 paend; struct scatterlist *sg; - struct device *dma = mvdev->mdev->device; + struct device *dma = get_dma_device(mvdev); for (map = vhost_iotlb_itree_first(iotlb, mr->start, mr->end - 1); map; map = vhost_iotlb_itree_next(map, start, mr->end - 1)) { @@ -291,7 +296,7 @@ static int map_direct_mr(struct mlx5_vdpa_dev *mvdev, struct mlx5_vdpa_direct_mr static void unmap_direct_mr(struct mlx5_vdpa_dev *mvdev, struct mlx5_vdpa_direct_mr *mr) { - struct device *dma = mvdev->mdev->device; + struct device *dma = get_dma_device(mvdev); destroy_direct_mr(mvdev, mr); dma_unmap_sg_attrs(dma, mr->sg_head.sgl, mr->nsg, DMA_BIDIRECTIONAL, 0); From patchwork Thu Apr 8 09:10:45 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eli Cohen X-Patchwork-Id: 418029 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER, INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A226FC43611 for ; Thu, 8 Apr 2021 09:11:10 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 4B2BD61107 for ; Thu, 8 Apr 2021 09:11:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230457AbhDHJLS (ORCPT ); Thu, 8 Apr 2021 05:11:18 -0400 Received: from mail-bn8nam12on2075.outbound.protection.outlook.com ([40.107.237.75]:61792 "EHLO NAM12-BN8-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S230337AbhDHJLO (ORCPT ); Thu, 8 Apr 2021 05:11:14 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=UdKIzzvOu2pyZ0BGqg/QYrsxwY0WQipiIkFhSLDeO9YKrmQLEx5U2G151x3Z/P8sNIM97P6X2+KiSAZY3XJ8DAEJFea9rIkipfDLSKexbumvHn8lI1Rn7k+aBQr9TYU1jf1HpM6ui+uHbJm0046yivLT41xbvY7mj/f6vEwSAyafhaWpBadUqefnfzEC2DbPNflcshDJZhqiK7HslOTkLekC7Dx/4HiXCHYp5fTlzhUt9czR5VCo7YUv/aZwy8WNUMGBGnB3cwB8oYsHzeJIhXD3P3lmsx/sgr1hzD4FVmVY4emv1l0WnPovFbUYmagnZEc1roptLb4FS96cAYWL5w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=qUgRl9uLOAK4/xuQCxY5O1ZGYVGQ5zGngOpRYaNnMdA=; b=Lv9LZN9FRA0E5q3Fq1nGWLqwdL5p8hn9yNjdexjIgDzpU3TxqcJXPvkMGF+RQKGfrVrGrH58mWYmG4OPJCrknVUm716VSf0km8azEHwdZiN99GJa98K3oMcFOQZau5SDRDe9lBInU2Lh2L52OstKIOV+LgpH49S28NpOiavr6XYUofIWNGFuBApaV847RJOsd2wZwRudjev1Z6XXz8bd61aM8zd8N9eqC5uFKdHin4sG7YvNS8u++yTWCNBc1v1MVDSKWnqWszeX6i/7QJOkNdLCyKkrcuCaK4ErOdMcloOTKftv/fDWh0cZFj7eTt20qAB7QRAQdcZDtepN1rNs9A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.36) smtp.rcpttodomain=lists.linux-foundation.org smtp.mailfrom=nvidia.com; dmarc=pass (p=none sp=none pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=qUgRl9uLOAK4/xuQCxY5O1ZGYVGQ5zGngOpRYaNnMdA=; b=YhzzaomF6+SgiOO57lLXeamgGd536wOxdgBvkfY/jXRmmEo4v7RdhhK16lni54ihIKhT9dXHMurnLWORCOD4O+4O8jYynJBWnARwaA05UHwsZnltGw4DEC6NYyBRcsWKOeKe9/2c561q+rUcJaUkMDbVwMdbPNXLFYYVIbXx5dNENQSeLkXkawBSw2YNwwkeHgVvURyjGvwcNcQey5nC4RW9Ae96MoJBK0I17218KeNAjqZlHrAhcvNs5qd2/k6lEN7oAE1Bp+ZV1B01j8uTTkLH4ec4PrhOIdgQMMsY7GTDFNBTv5gUjJykIMRmFe0xIpM+nrD0p+FqJkDg8GB/Ew== Received: from MWHPR04CA0062.namprd04.prod.outlook.com (2603:10b6:300:6c::24) by CY4PR12MB1509.namprd12.prod.outlook.com (2603:10b6:910:8::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3999.27; Thu, 8 Apr 2021 09:11:01 +0000 Received: from CO1NAM11FT022.eop-nam11.prod.protection.outlook.com (2603:10b6:300:6c:cafe::da) by MWHPR04CA0062.outlook.office365.com (2603:10b6:300:6c::24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4020.16 via Frontend Transport; Thu, 8 Apr 2021 09:11:01 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.36) smtp.mailfrom=nvidia.com; lists.linux-foundation.org; dkim=none (message not signed) header.d=none; lists.linux-foundation.org; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.36 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.36; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.36) by CO1NAM11FT022.mail.protection.outlook.com (10.13.175.199) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4020.17 via Frontend Transport; Thu, 8 Apr 2021 09:11:01 +0000 Received: from HQMAIL105.nvidia.com (172.20.187.12) by HQMAIL101.nvidia.com (172.20.187.10) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Thu, 8 Apr 2021 09:11:00 +0000 Received: from vdi.nvidia.com (172.20.145.6) by mail.nvidia.com (172.20.187.12) with Microsoft SMTP Server id 15.0.1497.2 via Frontend Transport; Thu, 8 Apr 2021 09:10:58 +0000 From: Eli Cohen To: , , , , , CC: , Eli Cohen Subject: [PATCH 3/5] vdpa/mlx5: Retrieve BAR address suitable any function Date: Thu, 8 Apr 2021 12:10:45 +0300 Message-ID: <20210408091047.4269-4-elic@nvidia.com> X-Mailer: git-send-email 2.30.1 In-Reply-To: <20210408091047.4269-1-elic@nvidia.com> References: <20210408091047.4269-1-elic@nvidia.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 719d1bb8-49c1-475d-4cde-08d8fa6e3ace X-MS-TrafficTypeDiagnostic: CY4PR12MB1509: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:4303; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: Mr+iuCG5lt2dCccUHChwUG7Rxf9c1jin8WBYocFI/hkn27e+KQq2AHVht0EHpjCUOwLfL/JorBEBlZ74D0nTbYMXImrWJbXGL2OlqSLDO9Xvo5jYAxOQlrU0562r0BkYlMqZkc6eDbk2E25f0hwICx7w7/9vR1L4p0MxpCcCEMxGYsQE6wyyEOnC7sig3OvvRwrAJO3/aQT5oTEF3kmtx+8TFDGwPyTlAaZormmlfm9QNqToZvT+33wcERcTCfHU2lmUCkczOICU4noiGh2PFWSSJDWpUHhbYQNIHLEI1dD83rbw0ZXJXqVY/BzvnXmMFwzHhDXVt4CxtdknmgpgOxoesj4e3ZFVxDOl4vmCYaTQ2z9CRpkyONn29bGQqplhIZ0SRimIN2Mlw8E85jG7cM3UomAGNzNcivEW4v+XS3n3cBxoK06+u+xO5DYTIkvtqI1aMAIn8cV3BsR0mL6w+pPpTxCFbRGVr00oupV7cqZ9ZNCNqD+ADtTMStxbbt3DotjjQavncrxErZzVHkhEaPMmKJ+/JzvEQHGvy0Y2Qe7YpafmfPNrpCDw/C1n8VvsfRIIWi3GVtQP0jbf9zsIFJ3jQja9QlCgTOreIGUmosE4fQtnYZ2F3/sDspEOKU4UxpfeFHVPv2o3l1aKB0n8L5X29mOeeLVxUc1sreLzdXM= X-Forefront-Antispam-Report: CIP:216.228.112.36; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:schybrid05.nvidia.com; CAT:NONE; SFS:(4636009)(346002)(376002)(39860400002)(396003)(136003)(36840700001)(46966006)(54906003)(110136005)(82740400003)(4744005)(107886003)(316002)(36906005)(8936002)(336012)(478600001)(83380400001)(426003)(356005)(7696005)(4326008)(1076003)(82310400003)(26005)(2906002)(7636003)(8676002)(36756003)(70206006)(36860700001)(6666004)(2616005)(186003)(5660300002)(70586007)(86362001)(47076005); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 08 Apr 2021 09:11:01.1830 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 719d1bb8-49c1-475d-4cde-08d8fa6e3ace X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.112.36]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CO1NAM11FT022.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY4PR12MB1509 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org struct mlx5_core_dev has a bar_addr field that contains the correct bar address for the function regardless of whether it is pci function or sub function. Use it. Fixes: 1958fc2f0712 ("net/mlx5: SF, Add auxiliary device driver") Signed-off-by: Eli Cohen Reviewed-by: Parav Pandit Acked-by: Jason Wang --- drivers/vdpa/mlx5/core/resources.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/drivers/vdpa/mlx5/core/resources.c b/drivers/vdpa/mlx5/core/resources.c index 96e6421c5d1c..6521cbd0f5c2 100644 --- a/drivers/vdpa/mlx5/core/resources.c +++ b/drivers/vdpa/mlx5/core/resources.c @@ -246,7 +246,8 @@ int mlx5_vdpa_alloc_resources(struct mlx5_vdpa_dev *mvdev) if (err) goto err_key; - kick_addr = pci_resource_start(mdev->pdev, 0) + offset; + kick_addr = mdev->bar_addr + offset; + res->kick_addr = ioremap(kick_addr, PAGE_SIZE); if (!res->kick_addr) { err = -ENOMEM; From patchwork Thu Apr 8 09:10:46 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eli Cohen X-Patchwork-Id: 418720 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER, INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id CC78BC43618 for ; Thu, 8 Apr 2021 09:11:11 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 775156112F for ; Thu, 8 Apr 2021 09:11:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230431AbhDHJLV (ORCPT ); Thu, 8 Apr 2021 05:11:21 -0400 Received: from mail-dm6nam11on2045.outbound.protection.outlook.com ([40.107.223.45]:18176 "EHLO NAM11-DM6-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S230397AbhDHJLR (ORCPT ); Thu, 8 Apr 2021 05:11:17 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=d4qDzHiv5Sz9e9umdFx4NIXAp7mFzi41ReJ5pC5LMD1Y7ara8fQoFrGZpxPJuwkpi2V7TAq/lxqAq78pAGf0V+NTCtVcn1o62jXbcZzu37V/6F3us2c3BvC/docEBaZ7r9i7SkCszv58GKFoGpJOdMA5dUy1QGpddqLGCp3WL9CrHj3BoeN4T9ke8sE+VkHSKMNQ77DX5vVzeJ9e5+CNQxVhFnyLC1vPYu8EbBEgK4K4ehi6wm4R4ofUuNJp6GjtXUNMalXJh5/nPyVDfLlmR1gEtHrFsVMPXLQfMnyK9QKSzRr6s+3gPFBF366pkoRlEgph6/LT4A+lIfYtpwCGUA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=4Coptz7DuRbmfq6qEdxYioO0mT5ALK1Kx9sK7KD7Pso=; b=PeVylaeu/BC0KgT8J0ibqdj8bxNBJhYKAZCqJtUe1e3A7Bfr8BuJqy+Hn962gWtDBqiq8VOQf30yhZgh1RoBZqoYOzPAcLC1dinJBlvP2Ey9/orb1TvL9Ue3atQV+LfRVjkytpF9l+qFmgAz9a0ooPTCgtz28nJ1FUhhc+zhAIjUIjh7VWLPwAt7O0o1hthUzhPbwRkK/9T4cbw3F5yQWBtYZKxgBdNTY/ShXBTNvkNieGwrDWIOO12qgDg/ymKOFjgL3osYGwPpPRbbneb32LEOWn6u9zcuzaNeBkMv1XIaUblOIARH+W8xywdyCesXgMdWUbvzxBRqQ7C8Pahlxw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.32) smtp.rcpttodomain=lists.linux-foundation.org smtp.mailfrom=nvidia.com; dmarc=pass (p=none sp=none pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=4Coptz7DuRbmfq6qEdxYioO0mT5ALK1Kx9sK7KD7Pso=; b=AH3DQgW6dRC1RDUd7fBcW+BQ+OTMG0LJjH9N8lJlFJyk21wXRaX7JDeF6vEpaOWIvp2dlfPCPrUdCODGO7TrKH5QNfc7p5vcJMAiguQGzTOBLZTkigmwnQ6auyHN/XMotd2PEsi46RUODpbHdlRsd+T/dbWD9715QbW5zziflGsF4hASOciL94yEBA38JS72fL9BULEO55taruKaCWmOGmyKkMQcXYVr1uM2Sh6fsm6thPFLj0GzoI1Tyo/srypO8k/0cJpzWV2Mx+962N3juQ+tcVKEt8WCzV9o0HEUEAIRo6bTETcvoQpgkBGJuWkscEVbD+MD1ztGTc6GKZqZeg== Received: from MWHPR18CA0048.namprd18.prod.outlook.com (2603:10b6:320:31::34) by BYAPR12MB3525.namprd12.prod.outlook.com (2603:10b6:a03:13b::26) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3999.29; Thu, 8 Apr 2021 09:11:05 +0000 Received: from CO1NAM11FT010.eop-nam11.prod.protection.outlook.com (2603:10b6:320:31:cafe::2f) by MWHPR18CA0048.outlook.office365.com (2603:10b6:320:31::34) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4020.16 via Frontend Transport; Thu, 8 Apr 2021 09:11:05 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.32) smtp.mailfrom=nvidia.com; lists.linux-foundation.org; dkim=none (message not signed) header.d=none; lists.linux-foundation.org; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.32 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.32; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.32) by CO1NAM11FT010.mail.protection.outlook.com (10.13.175.88) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4020.17 via Frontend Transport; Thu, 8 Apr 2021 09:11:03 +0000 Received: from HQMAIL105.nvidia.com (172.20.187.12) by HQMAIL109.nvidia.com (172.20.187.15) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Thu, 8 Apr 2021 02:11:03 -0700 Received: from vdi.nvidia.com (172.20.145.6) by mail.nvidia.com (172.20.187.12) with Microsoft SMTP Server id 15.0.1497.2 via Frontend Transport; Thu, 8 Apr 2021 09:11:01 +0000 From: Eli Cohen To: , , , , , CC: , Eli Cohen Subject: [PATCH 4/5] vdpa/mlx5: Fix wrong use of bit numbers Date: Thu, 8 Apr 2021 12:10:46 +0300 Message-ID: <20210408091047.4269-5-elic@nvidia.com> X-Mailer: git-send-email 2.30.1 In-Reply-To: <20210408091047.4269-1-elic@nvidia.com> References: <20210408091047.4269-1-elic@nvidia.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: fe604ead-f8c6-4477-ad04-08d8fa6e3c7f X-MS-TrafficTypeDiagnostic: BYAPR12MB3525: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:4125; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: S9ANR9COJB+K1NtfiWoGfm6slgroDC17YQ8gpilBzYCVYb2E4k7M85vZYWd7ed4GgA3eiC7BlV6o05+Wd3vTgiym0dApq3QlY/IFvEBstwQW9o2Z0m7Xs3B6kR6Ut/Y9rLD9B0P92ZU4xJPRxgCfg/vY1to1G2Qpcdn2OiIPC8hYF+ARXR45VgYcbRysi49kyArQzxkpAEAUybBmMOfSeCjFj5dD+CR0mzFugkac+XEuyvU1vezbqXbc6R1x7mGWBZo00bOeRbA0lvGjxA5rNAu2bs0uZSgO5mdVrO7GGL5R2xsAKYKJazl/IBwTAMJBXzXsWpTGLyhs9XCdLQmaCP9QmX6giAAe6dXhyAfaApmcE7bTSVhz3KINVbxm64Q2fX9MjqLigSMEhauz3DwL6XqtI/RoyW0dC9cVHObdKb2/jOO7hh+c0O6ySuA8cDq13gCFkn1aTBtY7gJvgCTUwCoGeZFVdsZ+UBGUqrzVblfOplTr6ttx1DbdEA5WGkspfaGMSZGurRNAZEbQGod4m0a4VEH7H08y3fJQPE7sMKQEWc6TTmM7C5WgJxDTEfJ9dLpT/HWRh0pN3aEffVCKVPms+25eL/gnxP234vUrg/tP6mGVIiyhveKuzrRK3htGEBicUGrSedV3HbZql3nqpTopvE6LmNZCNauTXPIbNyg= X-Forefront-Antispam-Report: CIP:216.228.112.32; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:schybrid01.nvidia.com; CAT:NONE; SFS:(4636009)(396003)(136003)(346002)(39860400002)(376002)(36840700001)(46966006)(47076005)(186003)(2616005)(36860700001)(1076003)(8676002)(83380400001)(110136005)(6666004)(5660300002)(36756003)(70206006)(26005)(70586007)(336012)(426003)(54906003)(316002)(2906002)(8936002)(7696005)(107886003)(478600001)(356005)(82310400003)(7636003)(86362001)(4326008)(82740400003); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 08 Apr 2021 09:11:03.9962 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: fe604ead-f8c6-4477-ad04-08d8fa6e3c7f X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.112.32]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CO1NAM11FT010.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR12MB3525 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org VIRTIO_F_VERSION_1 is a bit number. Use BIT_ULL() with mask conditionals. Also, in mlx5_vdpa_is_little_endian() use BIT_ULL for consistency with the rest of the code. Fixes: 1a86b377aa21 ("vdpa/mlx5: Add VDPA driver for supported mlx5 devices") Signed-off-by: Eli Cohen --- drivers/vdpa/mlx5/net/mlx5_vnet.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/drivers/vdpa/mlx5/net/mlx5_vnet.c b/drivers/vdpa/mlx5/net/mlx5_vnet.c index a49ebb250253..6fe61fc57790 100644 --- a/drivers/vdpa/mlx5/net/mlx5_vnet.c +++ b/drivers/vdpa/mlx5/net/mlx5_vnet.c @@ -820,7 +820,7 @@ static int create_virtqueue(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtque MLX5_SET(virtio_q, vq_ctx, event_qpn_or_msix, mvq->fwqp.mqp.qpn); MLX5_SET(virtio_q, vq_ctx, queue_size, mvq->num_ent); MLX5_SET(virtio_q, vq_ctx, virtio_version_1_0, - !!(ndev->mvdev.actual_features & VIRTIO_F_VERSION_1)); + !!(ndev->mvdev.actual_features & BIT_ULL(VIRTIO_F_VERSION_1))); MLX5_SET64(virtio_q, vq_ctx, desc_addr, mvq->desc_addr); MLX5_SET64(virtio_q, vq_ctx, used_addr, mvq->device_addr); MLX5_SET64(virtio_q, vq_ctx, available_addr, mvq->driver_addr); @@ -1554,7 +1554,7 @@ static void clear_virtqueues(struct mlx5_vdpa_net *ndev) static inline bool mlx5_vdpa_is_little_endian(struct mlx5_vdpa_dev *mvdev) { return virtio_legacy_is_little_endian() || - (mvdev->actual_features & (1ULL << VIRTIO_F_VERSION_1)); + (mvdev->actual_features & BIT_ULL(VIRTIO_F_VERSION_1)); } static __virtio16 cpu_to_mlx5vdpa16(struct mlx5_vdpa_dev *mvdev, u16 val) From patchwork Thu Apr 8 09:10:47 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eli Cohen X-Patchwork-Id: 418719 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER, INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2333EC43617 for ; Thu, 8 Apr 2021 09:11:13 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id B0D9961139 for ; Thu, 8 Apr 2021 09:11:12 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230488AbhDHJLW (ORCPT ); Thu, 8 Apr 2021 05:11:22 -0400 Received: from mail-co1nam11on2069.outbound.protection.outlook.com ([40.107.220.69]:42949 "EHLO NAM11-CO1-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S230449AbhDHJLT (ORCPT ); Thu, 8 Apr 2021 05:11:19 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=OyQGYlA0fAalmg9IlOQwR0D5B79D2DT0vYFeGgz7ZqY0P09fd564Ka2oIBrm+bUPu3YRwlDx2DihyCX3RJ4ViZ9K50Z9PonHPF0wtF8SSNNzM0jNyaLi5lAmjGvLskiEx5IBO04Q1JLXMM84odG663BcLW/LPkys9RUWGHUss2VkmDV+VLswjzXSeeaQEuEwwqJiH0jxX/Kt+9y2/v3Sq229B1dgDxx6nMFiTjWx2m26dhq7Jfw9OEHW+QBHKC1sg2XzJONgYcy9tfi93inHX4WinTB+TePUSftgfST52WEUOLfHwwIts3uDgEqP/xQXFdt3RDAoo5hZ9DlYLEdadQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=EUJPeEytsGu0gw6qqiqt0uGDmdO78mW7KcsR+Ob/Wxw=; b=Yx0xoyZXaBu3KWN3VGHynWqVSRQOyXw7tB845HsgMY9Sb9/SWlknbfMSsvi+WwnkUFQUYSCFMe9PN2fEG7/6A2c99ugCPJOCT8fjz5c51zKZKbA6FdrUrGCU6qI46rFTQpa4UqjrY7kzJfSm9NG3QK0fzsHn+kKXyKODpzUI/el3EbJ76/FXM91FGkKaKOelrHB1h8s2Z6yTl7nSu80B0pPiZZ8HNVyuLiyP2/Yzkw26LWlf/IkTWGHx+xNTIwWUWDAf+0m1blPyaz7cVcDP6m5/Kfu7PZVcshbPVjER4jehkSqNbVDj12KOhdAdv2ft+tGcciTkBZju12VU0g1DCg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.32) smtp.rcpttodomain=lists.linux-foundation.org smtp.mailfrom=nvidia.com; dmarc=pass (p=none sp=none pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=EUJPeEytsGu0gw6qqiqt0uGDmdO78mW7KcsR+Ob/Wxw=; b=nWhVfsCCUS9x7fuj8d3WtEbEKd1hYza+Z5fdMVUYltNx35h6GXbJO5PHieRsl1r1e2yTY/wSZiX6Pii5d1PvLgtRNCSsCIyl0/Fis2iV/eBMPEqbCqoP93eA90BKgKty2zkpZOYo/JSMyA8bINTjQlfbSXWmot65TGKdatOumZz4WFNGcvRW12RCTIo3X0BDy9j/9xWikr8AqLwFz+czuIt+TDOZxJ4GHowgm4B3g9Bp7d5BIUQAWRX/sjOmgHzU9Bi45V5Y0MlU4UTUTg2cf4aiGpcGy4CdeuQrWF36+DU0wpcqys3KWOuBP1Sgws6zcMb+ckw7eiGV3UyS2440Dw== Received: from MWHPR13CA0032.namprd13.prod.outlook.com (2603:10b6:300:95::18) by MWHPR12MB1262.namprd12.prod.outlook.com (2603:10b6:300:12::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3999.28; Thu, 8 Apr 2021 09:11:06 +0000 Received: from CO1NAM11FT012.eop-nam11.prod.protection.outlook.com (2603:10b6:300:95:cafe::b9) by MWHPR13CA0032.outlook.office365.com (2603:10b6:300:95::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4042.6 via Frontend Transport; Thu, 8 Apr 2021 09:11:06 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.32) smtp.mailfrom=nvidia.com; lists.linux-foundation.org; dkim=none (message not signed) header.d=none; lists.linux-foundation.org; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.32 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.32; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.32) by CO1NAM11FT012.mail.protection.outlook.com (10.13.175.192) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4020.17 via Frontend Transport; Thu, 8 Apr 2021 09:11:06 +0000 Received: from HQMAIL105.nvidia.com (172.20.187.12) by HQMAIL109.nvidia.com (172.20.187.15) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Thu, 8 Apr 2021 02:11:05 -0700 Received: from vdi.nvidia.com (172.20.145.6) by mail.nvidia.com (172.20.187.12) with Microsoft SMTP Server id 15.0.1497.2 via Frontend Transport; Thu, 8 Apr 2021 09:11:03 +0000 From: Eli Cohen To: , , , , , CC: , Eli Cohen Subject: [PATCH 5/5] vdpa/mlx5: Fix suspend/resume index restoration Date: Thu, 8 Apr 2021 12:10:47 +0300 Message-ID: <20210408091047.4269-6-elic@nvidia.com> X-Mailer: git-send-email 2.30.1 In-Reply-To: <20210408091047.4269-1-elic@nvidia.com> References: <20210408091047.4269-1-elic@nvidia.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: d9492701-4250-46ba-4545-08d8fa6e3dd5 X-MS-TrafficTypeDiagnostic: MWHPR12MB1262: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:3968; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: bSzaUJEl8/o9Q3uOciYOoJ8/aRO1B96V+KbeqEzUe6hp49y7ljF/8r+1Idi83DEiKApGUlvi8nGR+38HgS5VyGb2Z0rNoX0e7TyHUrw2NClkgdOrDHhZ/YvgUmi5xdvWPJOduNyJgJYWOgUjNcpTg0528zwpnnY1YCNDMOjFMjf0EXBogaBn/cnhqPyWLR6K4cIcm7bPEvlewlMMO2+o7TrBT3mFBGQdearwPmb2+2jBdjRh9wZRFOmFn8sPuWcf0S9TJKQq/4hvuZccUXMkwpWNItQeK3BsM6LuBeCls99Z5sJB6+SA5dsr+orSV0J4Rm0lyhHLeUfvKLFUj5GhtQhNi4oUbobgTHhCzzuamKU7Y64wmVqX/0QwuyEZ4T5AKRuBVmAfyArYP0O1cBIxUCQn6wPO1PZcokbrK8dna3cYHWHYr/38HBRXJM/3dc/Ci9VUjPRmV44G4ahH4UI3TR1yTGzukpDfAveBQhvVgL2TusWj5dOlqsqnT4l7mFtIuzZavAxIDN+NspIk18NSArk55E7/BWucZJi/9aiXzUoTsLK9AIZuhDMrzHMdMuAVeqnnN25EMnxLzaPCbt7oC4hCjnqenFfSoyxr8w63WB5dijsJMn+0C4zndCUI+YSaCvC3pgbdu5rZlJh6fanzSkJwHVkBRtnfqkmzoU+5nqI= X-Forefront-Antispam-Report: CIP:216.228.112.32; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:schybrid01.nvidia.com; CAT:NONE; SFS:(4636009)(39860400002)(346002)(376002)(396003)(136003)(36840700001)(46966006)(26005)(478600001)(8936002)(2906002)(8676002)(70586007)(47076005)(4326008)(86362001)(316002)(7636003)(6666004)(83380400001)(5660300002)(186003)(70206006)(7696005)(426003)(336012)(2616005)(1076003)(36756003)(82310400003)(107886003)(36860700001)(15650500001)(356005)(110136005)(54906003)(82740400003); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 08 Apr 2021 09:11:06.2389 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: d9492701-4250-46ba-4545-08d8fa6e3dd5 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.112.32]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CO1NAM11FT012.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR12MB1262 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org When we suspend the VM, the VDPA interface will be reset. When the VM is resumed again, clear_virtqueues() will clear the available and used indices resulting in hardware virqtqueue objects becoming out of sync. We can avoid this function alltogether since qemu will clear them if required, e.g. when the VM went through a reboot. Moreover, since the hw available and used indices should always be identical on query and should be restored to the same value same value for virtqueues that complete in order, we set the single value provided by set_vq_state(). In get_vq_state() we return the value of hardware used index. Fixes: b35ccebe3ef7 ("vdpa/mlx5: Restore the hardware used index after change map") Fixes: 1a86b377aa21 ("vdpa/mlx5: Add VDPA driver for supported mlx5 devices") Signed-off-by: Eli Cohen --- drivers/vdpa/mlx5/net/mlx5_vnet.c | 21 ++++++++------------- 1 file changed, 8 insertions(+), 13 deletions(-) diff --git a/drivers/vdpa/mlx5/net/mlx5_vnet.c b/drivers/vdpa/mlx5/net/mlx5_vnet.c index 6fe61fc57790..4d2809c7d4e3 100644 --- a/drivers/vdpa/mlx5/net/mlx5_vnet.c +++ b/drivers/vdpa/mlx5/net/mlx5_vnet.c @@ -1169,6 +1169,7 @@ static void suspend_vq(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtqueue *m return; } mvq->avail_idx = attr.available_index; + mvq->used_idx = attr.used_index; } static void suspend_vqs(struct mlx5_vdpa_net *ndev) @@ -1426,6 +1427,7 @@ static int mlx5_vdpa_set_vq_state(struct vdpa_device *vdev, u16 idx, return -EINVAL; } + mvq->used_idx = state->avail_index; mvq->avail_idx = state->avail_index; return 0; } @@ -1443,7 +1445,11 @@ static int mlx5_vdpa_get_vq_state(struct vdpa_device *vdev, u16 idx, struct vdpa * that cares about emulating the index after vq is stopped. */ if (!mvq->initialized) { - state->avail_index = mvq->avail_idx; + /* Firmware returns a wrong value for the available index. + * Since both values should be identical, we take the value of + * used_idx which is reported correctly. + */ + state->avail_index = mvq->used_idx; return 0; } @@ -1452,7 +1458,7 @@ static int mlx5_vdpa_get_vq_state(struct vdpa_device *vdev, u16 idx, struct vdpa mlx5_vdpa_warn(mvdev, "failed to query virtqueue\n"); return err; } - state->avail_index = attr.available_index; + state->avail_index = attr.used_index; return 0; } @@ -1540,16 +1546,6 @@ static void teardown_virtqueues(struct mlx5_vdpa_net *ndev) } } -static void clear_virtqueues(struct mlx5_vdpa_net *ndev) -{ - int i; - - for (i = ndev->mvdev.max_vqs - 1; i >= 0; i--) { - ndev->vqs[i].avail_idx = 0; - ndev->vqs[i].used_idx = 0; - } -} - /* TODO: cross-endian support */ static inline bool mlx5_vdpa_is_little_endian(struct mlx5_vdpa_dev *mvdev) { @@ -1785,7 +1781,6 @@ static void mlx5_vdpa_set_status(struct vdpa_device *vdev, u8 status) if (!status) { mlx5_vdpa_info(mvdev, "performing device reset\n"); teardown_driver(ndev); - clear_virtqueues(ndev); mlx5_vdpa_destroy_mr(&ndev->mvdev); ndev->mvdev.status = 0; ndev->mvdev.mlx_features = 0;