From patchwork Mon Mar 1 11:18:11 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vladimir Oltean X-Patchwork-Id: 389023 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 56295C433DB for ; Mon, 1 Mar 2021 11:19:06 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 137E564DF1 for ; Mon, 1 Mar 2021 11:19:06 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232269AbhCALS5 (ORCPT ); Mon, 1 Mar 2021 06:18:57 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54758 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232045AbhCALSw (ORCPT ); Mon, 1 Mar 2021 06:18:52 -0500 Received: from mail-ed1-x532.google.com (mail-ed1-x532.google.com [IPv6:2a00:1450:4864:20::532]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 982D1C061793 for ; Mon, 1 Mar 2021 03:18:36 -0800 (PST) Received: by mail-ed1-x532.google.com with SMTP id f6so4380523edd.12 for ; Mon, 01 Mar 2021 03:18:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=PNpEc5UnQICrqSCHItm3V5PWvmtOlQcEydqMRVoKzEI=; b=uTmZq7DVx3QsURIF1zwTJ7lR+pOLuRYOKGM3nehO9xSt+asD3tbNZIodZ8GdVohv38 Cl99Fe1ndffgM6GUO6a+JVNUD+hlYciDsalneaG3g8XuTcxJi30k5mqd28sdoRjHVmCH IX67uC89fELmGPr9zzHwsfiK33d3GdRYap044Kj102vkEctNdZviFeU/RZYFwMsCWlCy I/ySuwF8JEH4gfKW5iBU68w2DdxXnTdEptNAzVmFPqXvaMH846v+s0sJBDgjlYSdjM+o Ps5lZqu+lRkhwfSH3ZEmQH2RbEV9WLozHpSIdquk40+JJjXtVNSKxnMN2ii1pJWLTqO+ o17g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=PNpEc5UnQICrqSCHItm3V5PWvmtOlQcEydqMRVoKzEI=; b=p2yTATEe/Q2V1C3lFlX7092YRIns6cHM/9RvrLbglbimmmd9uH7vs3yPr2Lpqp5/sc NMVi7YiimIRZkRACW9KdOVPXO7aXiJi4TsTm/12Dp3UI74fj6I+6ieY3KCrGbGfpcDvs YoV7sKnvkKvGLAd5RKRtD2/HwPxR1ne3t5iF0hUR0dlElEMVmlylbkG9lwOMkR5HqLgC huXRq5tSyyOYSzes7KjNJ/W7FjNMS62yhRHLrfbR+j/g/09LbcfxVg7tfC2u00ZWGe/Q /ki95UK4ZiL8p8gQj8aLItRyjAyAzVA1eMiogj0PJdvucTCEdtdnyO7xRQgP+CJB1sf4 dYxw== X-Gm-Message-State: AOAM531/NmIZM1sZbaxC9t4GlZO+gWgz4G8wblzR5OmJpClq7uWP8eJY Q6ugSJIPKM1z1fBvk9bS740= X-Google-Smtp-Source: ABdhPJxbkFTua5ZRkUTQWMViFVcJbWF2UBE+Mi/gQWmf0eBKkplrifzWKltUwgyhDlqtBM1AjThLaA== X-Received: by 2002:a05:6402:1455:: with SMTP id d21mr15467465edx.192.1614597515262; Mon, 01 Mar 2021 03:18:35 -0800 (PST) Received: from localhost.localdomain ([188.25.217.13]) by smtp.gmail.com with ESMTPSA id i13sm13586491ejj.2.2021.03.01.03.18.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 01 Mar 2021 03:18:34 -0800 (PST) From: Vladimir Oltean To: "David S . Miller" , Jakub Kicinski , netdev@vger.kernel.org Cc: Michael Walle , Claudiu Manoil , Alexandru Marginean , Andrew Lunn , Vladimir Oltean , Jesse Brandeburg Subject: [PATCH v3 net 1/8] net: enetc: don't overwrite the RSS indirection table when initializing Date: Mon, 1 Mar 2021 13:18:11 +0200 Message-Id: <20210301111818.2081582-2-olteanv@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210301111818.2081582-1-olteanv@gmail.com> References: <20210301111818.2081582-1-olteanv@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Vladimir Oltean After the blamed patch, all RX traffic gets hashed to CPU 0 because the hashing indirection table set up in: enetc_pf_probe -> enetc_alloc_si_resources -> enetc_configure_si -> enetc_setup_default_rss_table is overwritten later in: enetc_pf_probe -> enetc_init_port_rss_memory which zero-initializes the entire port RSS table in order to avoid ECC errors. The trouble really is that enetc_init_port_rss_memory really neads enetc_alloc_si_resources to be called, because it depends upon enetc_alloc_cbdr and enetc_setup_cbdr. But that whole enetc_configure_si thing could have been better thought out, it has nothing to do in a function called "alloc_si_resources", especially since its counterpart, "free_si_resources", does nothing to unwind the configuration of the SI. The point is, we need to pull out enetc_configure_si out of enetc_alloc_resources, and move it after enetc_init_port_rss_memory. This allows us to set up the default RSS indirection table after initializing the memory. Fixes: 07bf34a50e32 ("net: enetc: initialize the RFS and RSS memories") Cc: Jesse Brandeburg Signed-off-by: Vladimir Oltean --- Changes in v3: None. Changes in v2: None. drivers/net/ethernet/freescale/enetc/enetc.c | 11 +++-------- drivers/net/ethernet/freescale/enetc/enetc.h | 1 + drivers/net/ethernet/freescale/enetc/enetc_pf.c | 7 +++++++ drivers/net/ethernet/freescale/enetc/enetc_vf.c | 7 +++++++ 4 files changed, 18 insertions(+), 8 deletions(-) diff --git a/drivers/net/ethernet/freescale/enetc/enetc.c b/drivers/net/ethernet/freescale/enetc/enetc.c index c78d12229730..fdb6b9e8da78 100644 --- a/drivers/net/ethernet/freescale/enetc/enetc.c +++ b/drivers/net/ethernet/freescale/enetc/enetc.c @@ -1058,13 +1058,12 @@ static int enetc_setup_default_rss_table(struct enetc_si *si, int num_groups) return 0; } -static int enetc_configure_si(struct enetc_ndev_priv *priv) +int enetc_configure_si(struct enetc_ndev_priv *priv) { struct enetc_si *si = priv->si; struct enetc_hw *hw = &si->hw; int err; - enetc_setup_cbdr(hw, &si->cbd_ring); /* set SI cache attributes */ enetc_wr(hw, ENETC_SICAR0, ENETC_SICAR_RD_COHERENT | ENETC_SICAR_WR_COHERENT); @@ -1112,6 +1111,8 @@ int enetc_alloc_si_resources(struct enetc_ndev_priv *priv) if (err) return err; + enetc_setup_cbdr(&si->hw, &si->cbd_ring); + priv->cls_rules = kcalloc(si->num_fs_entries, sizeof(*priv->cls_rules), GFP_KERNEL); if (!priv->cls_rules) { @@ -1119,14 +1120,8 @@ int enetc_alloc_si_resources(struct enetc_ndev_priv *priv) goto err_alloc_cls; } - err = enetc_configure_si(priv); - if (err) - goto err_config_si; - return 0; -err_config_si: - kfree(priv->cls_rules); err_alloc_cls: enetc_clear_cbdr(&si->hw); enetc_free_cbdr(priv->dev, &si->cbd_ring); diff --git a/drivers/net/ethernet/freescale/enetc/enetc.h b/drivers/net/ethernet/freescale/enetc/enetc.h index 8532d23b54f5..f8275cef3b5c 100644 --- a/drivers/net/ethernet/freescale/enetc/enetc.h +++ b/drivers/net/ethernet/freescale/enetc/enetc.h @@ -292,6 +292,7 @@ void enetc_get_si_caps(struct enetc_si *si); void enetc_init_si_rings_params(struct enetc_ndev_priv *priv); int enetc_alloc_si_resources(struct enetc_ndev_priv *priv); void enetc_free_si_resources(struct enetc_ndev_priv *priv); +int enetc_configure_si(struct enetc_ndev_priv *priv); int enetc_open(struct net_device *ndev); int enetc_close(struct net_device *ndev); diff --git a/drivers/net/ethernet/freescale/enetc/enetc_pf.c b/drivers/net/ethernet/freescale/enetc/enetc_pf.c index 515c5b29d7aa..d02ecb2e46ae 100644 --- a/drivers/net/ethernet/freescale/enetc/enetc_pf.c +++ b/drivers/net/ethernet/freescale/enetc/enetc_pf.c @@ -1108,6 +1108,12 @@ static int enetc_pf_probe(struct pci_dev *pdev, goto err_init_port_rss; } + err = enetc_configure_si(priv); + if (err) { + dev_err(&pdev->dev, "Failed to configure SI\n"); + goto err_config_si; + } + err = enetc_alloc_msix(priv); if (err) { dev_err(&pdev->dev, "MSIX alloc failed\n"); @@ -1136,6 +1142,7 @@ static int enetc_pf_probe(struct pci_dev *pdev, enetc_mdiobus_destroy(pf); err_mdiobus_create: enetc_free_msix(priv); +err_config_si: err_init_port_rss: err_init_port_rfs: err_alloc_msix: diff --git a/drivers/net/ethernet/freescale/enetc/enetc_vf.c b/drivers/net/ethernet/freescale/enetc/enetc_vf.c index 39c1a09e69a9..9b755a84c2d6 100644 --- a/drivers/net/ethernet/freescale/enetc/enetc_vf.c +++ b/drivers/net/ethernet/freescale/enetc/enetc_vf.c @@ -171,6 +171,12 @@ static int enetc_vf_probe(struct pci_dev *pdev, goto err_alloc_si_res; } + err = enetc_configure_si(priv); + if (err) { + dev_err(&pdev->dev, "Failed to configure SI\n"); + goto err_config_si; + } + err = enetc_alloc_msix(priv); if (err) { dev_err(&pdev->dev, "MSIX alloc failed\n"); @@ -187,6 +193,7 @@ static int enetc_vf_probe(struct pci_dev *pdev, err_reg_netdev: enetc_free_msix(priv); +err_config_si: err_alloc_msix: enetc_free_si_resources(priv); err_alloc_si_res: From patchwork Mon Mar 1 11:18:12 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vladimir Oltean X-Patchwork-Id: 390774 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id ED93EC433E0 for ; Mon, 1 Mar 2021 11:19:49 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 94A7C64DF5 for ; Mon, 1 Mar 2021 11:19:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232323AbhCALTp (ORCPT ); Mon, 1 Mar 2021 06:19:45 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54872 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232124AbhCALTR (ORCPT ); Mon, 1 Mar 2021 06:19:17 -0500 Received: from mail-ed1-x530.google.com (mail-ed1-x530.google.com [IPv6:2a00:1450:4864:20::530]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7C09CC061756 for ; Mon, 1 Mar 2021 03:18:37 -0800 (PST) Received: by mail-ed1-x530.google.com with SMTP id h10so20095079edl.6 for ; Mon, 01 Mar 2021 03:18:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=n/3TB3LAlq1rQiWc+V07/zAlcw6bhT/udy9/FkJs4Vc=; b=SxzwiOqJaw6PNXUcSXlCvW2HiHkHpkVaBla4SmmPTIjjH305AWFHpV4udG86ezea00 nZVwR+DUBN8H6huQQ7o9MOAqE4t6k0qriqtXpHENJvLRteEZ4yQJP9sLnsufJ/DIOsMR 4nGVBfPl/KNewa/InIt6bBZgVmyGMZ5WK46HKLZAlIPmOS+Nmlu7dr2CEeUC3oUt1gZ1 +RKn9Wz//ZQBRow905OhBTI2RqHtUhfUAym6b9K2n5RtVARHgU5YCJXK6FOwzyCtHnMD 5SeDJn/E4eVyeIOXar32NH9ZMhK9E1G+VETj89V40Ool0Hwm3azbrDRKQrMnmSoo0ErF NlyQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=n/3TB3LAlq1rQiWc+V07/zAlcw6bhT/udy9/FkJs4Vc=; b=IS7SIm9L7BF5U2EMd81fwgBGkTvSAzps33VozByDfGODGWjKljRVtjIlBtqxa81ZdF NAo8AGPMtllzREe7Et9PDVhrQc68ajLN+1SKQYgDDjwMFMqRRGIrBN2DIDbC1j+GQtvx foIGAbemKZc4JIQonT184F7h1oKWcptJVrbhIEZVKSgGffS4NvXhSbYLGzBZQW8ibyQ8 AAbSHP7FUDfogG3KjIEoB4/+lcYejWyQXi4qy6P3Z99Xb8wiL5pLY/x3qeXtPYO22SDv 6izD5uqIPxHZOjh0ldAE92jxCBc0f11p2JtQYnQ7pj0GpJxv7VuXEeMw0HUGKSFofRZh P4JA== X-Gm-Message-State: AOAM532Yu//oVAEpiTY/rs40aQp2W6HILcY4jx4jYp6ZJ5rrzCBQGHuy 7JSOCO5YzXHSvalpC4ZBrt8qZ0uwkak= X-Google-Smtp-Source: ABdhPJwFYVPJJSb0qB+N1NcKJWTtRsBW7AD9O5wRUCU5u13vs58wIUynSe8MdhiKfwgNEmBerbBSGQ== X-Received: by 2002:aa7:cb8e:: with SMTP id r14mr15911396edt.331.1614597516284; Mon, 01 Mar 2021 03:18:36 -0800 (PST) Received: from localhost.localdomain ([188.25.217.13]) by smtp.gmail.com with ESMTPSA id i13sm13586491ejj.2.2021.03.01.03.18.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 01 Mar 2021 03:18:35 -0800 (PST) From: Vladimir Oltean To: "David S . Miller" , Jakub Kicinski , netdev@vger.kernel.org Cc: Michael Walle , Claudiu Manoil , Alexandru Marginean , Andrew Lunn , Vladimir Oltean , Jesse Brandeburg Subject: [PATCH v3 net 2/8] net: enetc: initialize RFS/RSS memories for unused ports too Date: Mon, 1 Mar 2021 13:18:12 +0200 Message-Id: <20210301111818.2081582-3-olteanv@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210301111818.2081582-1-olteanv@gmail.com> References: <20210301111818.2081582-1-olteanv@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Vladimir Oltean Michael reports that since linux-next-20210211, the AER messages for ECC errors have started reappearing, and this time they can be reliably reproduced with the first ping on one of his LS1028A boards. $ ping 1[ 33.258069] pcieport 0000:00:1f.0: AER: Multiple Corrected error received: 0000:00:00.0 72.16.0.1 PING [ 33.267050] pcieport 0000:00:1f.0: AER: can't find device of ID0000 172.16.0.1 (172.16.0.1): 56 data bytes 64 bytes from 172.16.0.1: seq=0 ttl=64 time=17.124 ms 64 bytes from 172.16.0.1: seq=1 ttl=64 time=0.273 ms $ devmem 0x1f8010e10 32 0xC0000006 It isn't clear why this is necessary, but it seems that for the errors to go away, we must clear the entire RFS and RSS memory, not just for the ports in use. Sadly the code is structured in such a way that we can't have unified logic for the used and unused ports. For the minimal initialization of an unused port, we need just to enable and ioremap the PF memory space, and a control buffer descriptor ring. Unused ports must then free the CBDR because the driver will exit, but used ports can not pick up from where that code path left, since the CBDR API does not reinitialize a ring when setting it up, so its producer and consumer indices are out of sync between the software and hardware state. So a separate enetc_init_unused_port function was created, and it gets called right after the PF memory space is enabled. Fixes: 07bf34a50e32 ("net: enetc: initialize the RFS and RSS memories") Reported-by: Michael Walle Cc: Jesse Brandeburg Signed-off-by: Vladimir Oltean Tested-by: Michael Walle --- Changes in v3: Leave the moving around of cbdr functions to a later refactoring patch, and keep the bug fix minimal. Changes in v2: None. drivers/net/ethernet/freescale/enetc/enetc.c | 8 ++--- drivers/net/ethernet/freescale/enetc/enetc.h | 4 +++ .../net/ethernet/freescale/enetc/enetc_pf.c | 33 ++++++++++++++++--- 3 files changed, 36 insertions(+), 9 deletions(-) diff --git a/drivers/net/ethernet/freescale/enetc/enetc.c b/drivers/net/ethernet/freescale/enetc/enetc.c index fdb6b9e8da78..eb45830a1667 100644 --- a/drivers/net/ethernet/freescale/enetc/enetc.c +++ b/drivers/net/ethernet/freescale/enetc/enetc.c @@ -984,7 +984,7 @@ static void enetc_free_rxtx_rings(struct enetc_ndev_priv *priv) enetc_free_tx_ring(priv->tx_ring[i]); } -static int enetc_alloc_cbdr(struct device *dev, struct enetc_cbdr *cbdr) +int enetc_alloc_cbdr(struct device *dev, struct enetc_cbdr *cbdr) { int size = cbdr->bd_count * sizeof(struct enetc_cbd); @@ -1005,7 +1005,7 @@ static int enetc_alloc_cbdr(struct device *dev, struct enetc_cbdr *cbdr) return 0; } -static void enetc_free_cbdr(struct device *dev, struct enetc_cbdr *cbdr) +void enetc_free_cbdr(struct device *dev, struct enetc_cbdr *cbdr) { int size = cbdr->bd_count * sizeof(struct enetc_cbd); @@ -1013,7 +1013,7 @@ static void enetc_free_cbdr(struct device *dev, struct enetc_cbdr *cbdr) cbdr->bd_base = NULL; } -static void enetc_setup_cbdr(struct enetc_hw *hw, struct enetc_cbdr *cbdr) +void enetc_setup_cbdr(struct enetc_hw *hw, struct enetc_cbdr *cbdr) { /* set CBDR cache attributes */ enetc_wr(hw, ENETC_SICAR2, @@ -1033,7 +1033,7 @@ static void enetc_setup_cbdr(struct enetc_hw *hw, struct enetc_cbdr *cbdr) cbdr->cir = hw->reg + ENETC_SICBDRCIR; } -static void enetc_clear_cbdr(struct enetc_hw *hw) +void enetc_clear_cbdr(struct enetc_hw *hw) { enetc_wr(hw, ENETC_SICBDRMR, 0); } diff --git a/drivers/net/ethernet/freescale/enetc/enetc.h b/drivers/net/ethernet/freescale/enetc/enetc.h index f8275cef3b5c..8b380fc13314 100644 --- a/drivers/net/ethernet/freescale/enetc/enetc.h +++ b/drivers/net/ethernet/freescale/enetc/enetc.h @@ -310,6 +310,10 @@ int enetc_setup_tc(struct net_device *ndev, enum tc_setup_type type, void enetc_set_ethtool_ops(struct net_device *ndev); /* control buffer descriptor ring (CBDR) */ +int enetc_alloc_cbdr(struct device *dev, struct enetc_cbdr *cbdr); +void enetc_free_cbdr(struct device *dev, struct enetc_cbdr *cbdr); +void enetc_setup_cbdr(struct enetc_hw *hw, struct enetc_cbdr *cbdr); +void enetc_clear_cbdr(struct enetc_hw *hw); int enetc_set_mac_flt_entry(struct enetc_si *si, int index, char *mac_addr, int si_map); int enetc_clear_mac_flt_entry(struct enetc_si *si, int index); diff --git a/drivers/net/ethernet/freescale/enetc/enetc_pf.c b/drivers/net/ethernet/freescale/enetc/enetc_pf.c index d02ecb2e46ae..62ba4bf56f0d 100644 --- a/drivers/net/ethernet/freescale/enetc/enetc_pf.c +++ b/drivers/net/ethernet/freescale/enetc/enetc_pf.c @@ -1041,6 +1041,26 @@ static int enetc_init_port_rss_memory(struct enetc_si *si) return err; } +static void enetc_init_unused_port(struct enetc_si *si) +{ + struct device *dev = &si->pdev->dev; + struct enetc_hw *hw = &si->hw; + int err; + + si->cbd_ring.bd_count = ENETC_CBDR_DEFAULT_SIZE; + err = enetc_alloc_cbdr(dev, &si->cbd_ring); + if (err) + return; + + enetc_setup_cbdr(hw, &si->cbd_ring); + + enetc_init_port_rfs_memory(si); + enetc_init_port_rss_memory(si); + + enetc_clear_cbdr(hw); + enetc_free_cbdr(dev, &si->cbd_ring); +} + static int enetc_pf_probe(struct pci_dev *pdev, const struct pci_device_id *ent) { @@ -1051,11 +1071,6 @@ static int enetc_pf_probe(struct pci_dev *pdev, struct enetc_pf *pf; int err; - if (node && !of_device_is_available(node)) { - dev_info(&pdev->dev, "device is disabled, skipping\n"); - return -ENODEV; - } - err = enetc_pci_probe(pdev, KBUILD_MODNAME, sizeof(*pf)); if (err) { dev_err(&pdev->dev, "PCI probing failed\n"); @@ -1069,6 +1084,13 @@ static int enetc_pf_probe(struct pci_dev *pdev, goto err_map_pf_space; } + if (node && !of_device_is_available(node)) { + enetc_init_unused_port(si); + dev_info(&pdev->dev, "device is disabled, skipping\n"); + err = -ENODEV; + goto err_device_disabled; + } + pf = enetc_si_priv(si); pf->si = si; pf->total_vfs = pci_sriov_get_totalvfs(pdev); @@ -1151,6 +1173,7 @@ static int enetc_pf_probe(struct pci_dev *pdev, si->ndev = NULL; free_netdev(ndev); err_alloc_netdev: +err_device_disabled: err_map_pf_space: enetc_pci_remove(pdev); From patchwork Mon Mar 1 11:18:13 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vladimir Oltean X-Patchwork-Id: 389021 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 276A8C433E0 for ; Mon, 1 Mar 2021 11:19:57 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id E78E264DF1 for ; Mon, 1 Mar 2021 11:19:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232400AbhCALTx (ORCPT ); Mon, 1 Mar 2021 06:19:53 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54874 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232139AbhCALTS (ORCPT ); Mon, 1 Mar 2021 06:19:18 -0500 Received: from mail-ej1-x62b.google.com (mail-ej1-x62b.google.com [IPv6:2a00:1450:4864:20::62b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 575A2C0617A9 for ; Mon, 1 Mar 2021 03:18:38 -0800 (PST) Received: by mail-ej1-x62b.google.com with SMTP id jt13so27514382ejb.0 for ; Mon, 01 Mar 2021 03:18:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=P2UgqtIaZNShOADApKa/iEfTtyfD4uVb4zvNzvGRm38=; b=G2iTemoXzbsNVbC3XR3ZXiIhGovsJCNisrJI1pqJr/Dv1YmRItqrsdI7/TQE2Er3k4 g4LQ1bQFMh1mWcqIJfNhthTguyHj3DJESL+5CzAB/lwFFnzevwKecS0sNaj+C+Xb07NG WqBUg6ptF3/08naurG+CIvA/Gd2mIW8Rwq3DfhlGmyAW6st3/c6JJ40cZGIyUJXJ48uM 4BNYadw5pZdcgCLrDfScWcJFloFr8MSlRI6IybAkDl8BgQlO5pgTtYvWux58GyYfCQnS Kg7ZU5dN5OaYTn1fpyNOXQ6CZFxlONUr0xFQix8pb4X5aEIQKlchI3Q+lW5KVyHgA5VV PbPg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=P2UgqtIaZNShOADApKa/iEfTtyfD4uVb4zvNzvGRm38=; b=UOD0+GvQuIazt4CvbOTe6Xesyxhad9djx6PyhwhWw74HvmKbmDFLr603A+HTRvmAb2 oEFReUIvrXLcgo4ysD6/c4bSRTa68SYjxmfqpvP1ZrooRxfbfhxb1Ai9gxGCMphTae2W EA1coEFNqEfmndxZWi1q+QA5fokgzjhwhYNqzkAU2XsuiYUDb+aXVWKE6ebgoM+kvUf2 9/wDLOrzaYLywAixqQWCtdBm2P7MU0nA2LdgGIaYQ1MJ+GjgZd5q+y0LYNzMLJS7wa0k EtBmg185VPqX7wFpwS5Al9gywkg6ceaCy8Yge9U7aXT2FZWC3a0JwZs2Ox7nnJxCIJr/ VErQ== X-Gm-Message-State: AOAM532DUdR56xO8dCqw5VQrycdnmVK3ZemHgX5tDE4xzmvU+ZHN+DwC t6u6QR59GFar22+/nfHi43gE/mHybcI= X-Google-Smtp-Source: ABdhPJzTYVVL9JJ4HtavY3b2JpIAklMSgdAgAbDC/DILSlh2tJT9JoSWbFw4IH7iT86pDeC2T2uaHQ== X-Received: by 2002:a17:906:229b:: with SMTP id p27mr14831233eja.287.1614597517135; Mon, 01 Mar 2021 03:18:37 -0800 (PST) Received: from localhost.localdomain ([188.25.217.13]) by smtp.gmail.com with ESMTPSA id i13sm13586491ejj.2.2021.03.01.03.18.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 01 Mar 2021 03:18:36 -0800 (PST) From: Vladimir Oltean To: "David S . Miller" , Jakub Kicinski , netdev@vger.kernel.org Cc: Michael Walle , Claudiu Manoil , Alexandru Marginean , Andrew Lunn , Vladimir Oltean Subject: [PATCH v3 net 3/8] net: enetc: take the MDIO lock only once per NAPI poll cycle Date: Mon, 1 Mar 2021 13:18:13 +0200 Message-Id: <20210301111818.2081582-4-olteanv@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210301111818.2081582-1-olteanv@gmail.com> References: <20210301111818.2081582-1-olteanv@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Vladimir Oltean The workaround for the ENETC MDIO erratum caused a performance degradation of 82 Kpps (seen with IP forwarding of two 1Gbps streams of 64B packets). This is due to excessive locking and unlocking in the fast path, which can be avoided. By taking the MDIO read-side lock only once per NAPI poll cycle, we are able to regain 54 Kpps (65%) of the performance hit. The rest of the performance degradation comes from the TX data path, but unfortunately it doesn't look like we can optimize that away easily, even with netdev_xmit_more(), there just isn't any skb batching done, to help with taking the MDIO lock less often than once per packet. We need to change the register accessor type for enetc_get_tx_tstamp, because it now runs under the enetc_lock_mdio as per the new call path detailed below: enetc_msix -> napi_schedule -> enetc_poll -> enetc_lock_mdio -> enetc_clean_tx_ring -> enetc_get_tx_tstamp -> enetc_clean_rx_ring -> enetc_unlock_mdio Fixes: fd5736bf9f23 ("enetc: Workaround for MDIO register access issue") Signed-off-by: Vladimir Oltean --- Changes in v3: Detailed the enetc_get_tx_tstamp change as per Andrew's request. Changes in v2: None. drivers/net/ethernet/freescale/enetc/enetc.c | 31 ++++++------------- .../net/ethernet/freescale/enetc/enetc_hw.h | 2 ++ 2 files changed, 11 insertions(+), 22 deletions(-) diff --git a/drivers/net/ethernet/freescale/enetc/enetc.c b/drivers/net/ethernet/freescale/enetc/enetc.c index eb45830a1667..9bcceb74fb9c 100644 --- a/drivers/net/ethernet/freescale/enetc/enetc.c +++ b/drivers/net/ethernet/freescale/enetc/enetc.c @@ -281,6 +281,8 @@ static int enetc_poll(struct napi_struct *napi, int budget) int work_done; int i; + enetc_lock_mdio(); + for (i = 0; i < v->count_tx_rings; i++) if (!enetc_clean_tx_ring(&v->tx_ring[i], budget)) complete = false; @@ -291,8 +293,10 @@ static int enetc_poll(struct napi_struct *napi, int budget) if (work_done) v->rx_napi_work = true; - if (!complete) + if (!complete) { + enetc_unlock_mdio(); return budget; + } napi_complete_done(napi, work_done); @@ -301,8 +305,6 @@ static int enetc_poll(struct napi_struct *napi, int budget) v->rx_napi_work = false; - enetc_lock_mdio(); - /* enable interrupts */ enetc_wr_reg_hot(v->rbier, ENETC_RBIER_RXTIE); @@ -327,8 +329,8 @@ static void enetc_get_tx_tstamp(struct enetc_hw *hw, union enetc_tx_bd *txbd, { u32 lo, hi, tstamp_lo; - lo = enetc_rd(hw, ENETC_SICTR0); - hi = enetc_rd(hw, ENETC_SICTR1); + lo = enetc_rd_hot(hw, ENETC_SICTR0); + hi = enetc_rd_hot(hw, ENETC_SICTR1); tstamp_lo = le32_to_cpu(txbd->wb.tstamp); if (lo <= tstamp_lo) hi -= 1; @@ -358,9 +360,7 @@ static bool enetc_clean_tx_ring(struct enetc_bdr *tx_ring, int napi_budget) i = tx_ring->next_to_clean; tx_swbd = &tx_ring->tx_swbd[i]; - enetc_lock_mdio(); bds_to_clean = enetc_bd_ready_count(tx_ring, i); - enetc_unlock_mdio(); do_tstamp = false; @@ -403,8 +403,6 @@ static bool enetc_clean_tx_ring(struct enetc_bdr *tx_ring, int napi_budget) tx_swbd = tx_ring->tx_swbd; } - enetc_lock_mdio(); - /* BD iteration loop end */ if (is_eof) { tx_frm_cnt++; @@ -415,8 +413,6 @@ static bool enetc_clean_tx_ring(struct enetc_bdr *tx_ring, int napi_budget) if (unlikely(!bds_to_clean)) bds_to_clean = enetc_bd_ready_count(tx_ring, i); - - enetc_unlock_mdio(); } tx_ring->next_to_clean = i; @@ -660,8 +656,6 @@ static int enetc_clean_rx_ring(struct enetc_bdr *rx_ring, u32 bd_status; u16 size; - enetc_lock_mdio(); - if (cleaned_cnt >= ENETC_RXBD_BUNDLE) { int count = enetc_refill_rx_ring(rx_ring, cleaned_cnt); @@ -672,19 +666,15 @@ static int enetc_clean_rx_ring(struct enetc_bdr *rx_ring, rxbd = enetc_rxbd(rx_ring, i); bd_status = le32_to_cpu(rxbd->r.lstatus); - if (!bd_status) { - enetc_unlock_mdio(); + if (!bd_status) break; - } enetc_wr_reg_hot(rx_ring->idr, BIT(rx_ring->index)); dma_rmb(); /* for reading other rxbd fields */ size = le16_to_cpu(rxbd->r.buf_len); skb = enetc_map_rx_buff_to_skb(rx_ring, i, size); - if (!skb) { - enetc_unlock_mdio(); + if (!skb) break; - } enetc_get_offloads(rx_ring, rxbd, skb); @@ -696,7 +686,6 @@ static int enetc_clean_rx_ring(struct enetc_bdr *rx_ring, if (unlikely(bd_status & ENETC_RXBD_LSTATUS(ENETC_RXBD_ERR_MASK))) { - enetc_unlock_mdio(); dev_kfree_skb(skb); while (!(bd_status & ENETC_RXBD_LSTATUS_F)) { dma_rmb(); @@ -736,8 +725,6 @@ static int enetc_clean_rx_ring(struct enetc_bdr *rx_ring, enetc_process_skb(rx_ring, skb); - enetc_unlock_mdio(); - napi_gro_receive(napi, skb); rx_frm_cnt++; diff --git a/drivers/net/ethernet/freescale/enetc/enetc_hw.h b/drivers/net/ethernet/freescale/enetc/enetc_hw.h index c71fe8d751d5..8b54562f5da6 100644 --- a/drivers/net/ethernet/freescale/enetc/enetc_hw.h +++ b/drivers/net/ethernet/freescale/enetc/enetc_hw.h @@ -453,6 +453,8 @@ static inline u64 _enetc_rd_reg64_wa(void __iomem *reg) #define enetc_wr_reg(reg, val) _enetc_wr_reg_wa((reg), (val)) #define enetc_rd(hw, off) enetc_rd_reg((hw)->reg + (off)) #define enetc_wr(hw, off, val) enetc_wr_reg((hw)->reg + (off), val) +#define enetc_rd_hot(hw, off) enetc_rd_reg_hot((hw)->reg + (off)) +#define enetc_wr_hot(hw, off, val) enetc_wr_reg_hot((hw)->reg + (off), val) #define enetc_rd64(hw, off) _enetc_rd_reg64_wa((hw)->reg + (off)) /* port register accessors - PF only */ #define enetc_port_rd(hw, off) enetc_rd_reg((hw)->port + (off)) From patchwork Mon Mar 1 11:18:14 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vladimir Oltean X-Patchwork-Id: 390773 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 85ABCC433DB for ; Mon, 1 Mar 2021 11:20:35 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 5A15564DE0 for ; Mon, 1 Mar 2021 11:20:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232471AbhCALUS (ORCPT ); Mon, 1 Mar 2021 06:20:18 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54880 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232098AbhCALTT (ORCPT ); Mon, 1 Mar 2021 06:19:19 -0500 Received: from mail-ed1-x532.google.com (mail-ed1-x532.google.com [IPv6:2a00:1450:4864:20::532]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 305C9C0617AA for ; Mon, 1 Mar 2021 03:18:39 -0800 (PST) Received: by mail-ed1-x532.google.com with SMTP id p1so15656156edy.2 for ; Mon, 01 Mar 2021 03:18:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=sULpWLe+NRtZFEp300sV7rIrU+posp1UnSxEOcxGHBo=; b=nNT3M0nlveqdqQcsCkwxQB1CBWBgrPTrlASo96B+xrFkm8X6/GCuLcjwptxXq+cN9K YUc4vNeNqvhmX/9FiEXhyVZTpstQKulZ9VUitw6GKqkJZObqfvusdnDdOPiLk8S4tQon 0AFHSyz47c3GseI/FvW6m0AOmXoIpOIBqNGeGuClp3jy2vO4mEJvqZFf+a9VnVluW4IZ w0SJZVN/7n7cYnHcgQ/N6x4FHqOwJ2q2Z0bEIUWQeSd4qcHffKHcDhrMWlTMvjaD6IiS t1dEoIj4Pd4UvGxevKroI2hFNwNfz9J2LLF2xpuD8NTn/M5MTXyoQyvPwIFJUmPaxTIL CmQA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=sULpWLe+NRtZFEp300sV7rIrU+posp1UnSxEOcxGHBo=; b=O7UVObDmxTGJyB0sYywCvTbO9pJXdWxjOvrZl2Zp+dIvHAX2cIvA5ulx5mt3ozjPpF cb/WGM/scz9NM1djJVsTiXbXj6qrYYciVb1XY4vQ2SzrnucGHqFx0VF9N+aGct6Kls/E LftCQZSGHhW2+WyZSC9CWDXIxpimhTbf/ep8847jwzc6g+wNUbpRn/flpbVPe3ncnmrN /6EBx7NT4JS4YtAdVxkEXjzXHZ0OxxK33at2nd710dnDVSYP/gSQfUk3VNrq6KcEtGz0 S7g8kksXHFStuZViyAa/gbYZE1jxUmw17s7NcJoDXrw6WxfttU12Zkc2uqzMU3B/1/OL OkIw== X-Gm-Message-State: AOAM532bXp7CCdG5I8/p2v0+weYvRyXCv8h1LrJfgb0qAuSu36XLQMlA IDS9LxVD0Lcc3uQZXY7gOQ8= X-Google-Smtp-Source: ABdhPJwpCSQB8GfQnkAx4Ws/o+6P0H9lVUoGv5KjrX18Yd5IxzDHAvIUGLVE4tMD5KFHesFLo2cJpQ== X-Received: by 2002:aa7:c456:: with SMTP id n22mr15685253edr.277.1614597518000; Mon, 01 Mar 2021 03:18:38 -0800 (PST) Received: from localhost.localdomain ([188.25.217.13]) by smtp.gmail.com with ESMTPSA id i13sm13586491ejj.2.2021.03.01.03.18.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 01 Mar 2021 03:18:37 -0800 (PST) From: Vladimir Oltean To: "David S . Miller" , Jakub Kicinski , netdev@vger.kernel.org Cc: Michael Walle , Claudiu Manoil , Alexandru Marginean , Andrew Lunn , Vladimir Oltean Subject: [PATCH v3 net 4/8] net: enetc: fix incorrect TPID when receiving 802.1ad tagged packets Date: Mon, 1 Mar 2021 13:18:14 +0200 Message-Id: <20210301111818.2081582-5-olteanv@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210301111818.2081582-1-olteanv@gmail.com> References: <20210301111818.2081582-1-olteanv@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Vladimir Oltean When the enetc ports have rx-vlan-offload enabled, they report a TPID of ETH_P_8021Q regardless of what was actually in the packet. When rx-vlan-offload is disabled, packets have the proper TPID. Fix this inconsistency by finishing the TODO left in the code. Fixes: d4fd0404c1c9 ("enetc: Introduce basic PF and VF ENETC ethernet drivers") Signed-off-by: Vladimir Oltean --- Changes in v3: None. Changes in v2: The "priv" variable needs to be used regardless of CONFIG_FSL_ENETC_PTP_CLOCK now. drivers/net/ethernet/freescale/enetc/enetc.c | 34 ++++++++++++++----- .../net/ethernet/freescale/enetc/enetc_hw.h | 3 ++ 2 files changed, 29 insertions(+), 8 deletions(-) diff --git a/drivers/net/ethernet/freescale/enetc/enetc.c b/drivers/net/ethernet/freescale/enetc/enetc.c index 9bcceb74fb9c..8ddf0cdc37a5 100644 --- a/drivers/net/ethernet/freescale/enetc/enetc.c +++ b/drivers/net/ethernet/freescale/enetc/enetc.c @@ -523,9 +523,8 @@ static void enetc_get_rx_tstamp(struct net_device *ndev, static void enetc_get_offloads(struct enetc_bdr *rx_ring, union enetc_rx_bd *rxbd, struct sk_buff *skb) { -#ifdef CONFIG_FSL_ENETC_PTP_CLOCK struct enetc_ndev_priv *priv = netdev_priv(rx_ring->ndev); -#endif + /* TODO: hashing */ if (rx_ring->ndev->features & NETIF_F_RXCSUM) { u16 inet_csum = le16_to_cpu(rxbd->r.inet_csum); @@ -534,12 +533,31 @@ static void enetc_get_offloads(struct enetc_bdr *rx_ring, skb->ip_summed = CHECKSUM_COMPLETE; } - /* copy VLAN to skb, if one is extracted, for now we assume it's a - * standard TPID, but HW also supports custom values - */ - if (le16_to_cpu(rxbd->r.flags) & ENETC_RXBD_FLAG_VLAN) - __vlan_hwaccel_put_tag(skb, htons(ETH_P_8021Q), - le16_to_cpu(rxbd->r.vlan_opt)); + if (le16_to_cpu(rxbd->r.flags) & ENETC_RXBD_FLAG_VLAN) { + __be16 tpid = 0; + + switch (le16_to_cpu(rxbd->r.flags) & ENETC_RXBD_FLAG_TPID) { + case 0: + tpid = htons(ETH_P_8021Q); + break; + case 1: + tpid = htons(ETH_P_8021AD); + break; + case 2: + tpid = htons(enetc_port_rd(&priv->si->hw, + ENETC_PCVLANR1)); + break; + case 3: + tpid = htons(enetc_port_rd(&priv->si->hw, + ENETC_PCVLANR2)); + break; + default: + break; + } + + __vlan_hwaccel_put_tag(skb, tpid, le16_to_cpu(rxbd->r.vlan_opt)); + } + #ifdef CONFIG_FSL_ENETC_PTP_CLOCK if (priv->active_offloads & ENETC_F_RX_TSTAMP) enetc_get_rx_tstamp(rx_ring->ndev, rxbd, skb); diff --git a/drivers/net/ethernet/freescale/enetc/enetc_hw.h b/drivers/net/ethernet/freescale/enetc/enetc_hw.h index 8b54562f5da6..a62604a1e54e 100644 --- a/drivers/net/ethernet/freescale/enetc/enetc_hw.h +++ b/drivers/net/ethernet/freescale/enetc/enetc_hw.h @@ -172,6 +172,8 @@ enum enetc_bdr_type {TX, RX}; #define ENETC_PSIPMAR0(n) (0x0100 + (n) * 0x8) /* n = SI index */ #define ENETC_PSIPMAR1(n) (0x0104 + (n) * 0x8) #define ENETC_PVCLCTR 0x0208 +#define ENETC_PCVLANR1 0x0210 +#define ENETC_PCVLANR2 0x0214 #define ENETC_VLAN_TYPE_C BIT(0) #define ENETC_VLAN_TYPE_S BIT(1) #define ENETC_PVCLCTR_OVTPIDL(bmp) ((bmp) & 0xff) /* VLAN_TYPE */ @@ -570,6 +572,7 @@ union enetc_rx_bd { #define ENETC_RXBD_LSTATUS(flags) ((flags) << 16) #define ENETC_RXBD_FLAG_VLAN BIT(9) #define ENETC_RXBD_FLAG_TSTMP BIT(10) +#define ENETC_RXBD_FLAG_TPID GENMASK(1, 0) #define ENETC_MAC_ADDR_FILT_CNT 8 /* # of supported entries per port */ #define EMETC_MAC_ADDR_FILT_RES 3 /* # of reserved entries at the beginning */ From patchwork Mon Mar 1 11:18:15 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Vladimir Oltean X-Patchwork-Id: 389020 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 74FFBC433DB for ; Mon, 1 Mar 2021 11:20:50 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 39F7464DF5 for ; Mon, 1 Mar 2021 11:20:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232362AbhCALUk (ORCPT ); Mon, 1 Mar 2021 06:20:40 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54886 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232268AbhCALTU (ORCPT ); Mon, 1 Mar 2021 06:19:20 -0500 Received: from mail-ed1-x531.google.com (mail-ed1-x531.google.com [IPv6:2a00:1450:4864:20::531]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2459DC0617AB for ; Mon, 1 Mar 2021 03:18:40 -0800 (PST) Received: by mail-ed1-x531.google.com with SMTP id c6so20190118ede.0 for ; Mon, 01 Mar 2021 03:18:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=buJA7JBdtnLfaYWGzMfKmcyZstmCAza5dcaiN9jkd8o=; b=E5BRthDNboKLvCnPz8qzS1KHvexDGOLIH5ktDX2FxzTLOMMrEQKRhmbXHQmcvSlTQy Ax/+UJ1nsxAtO3e2EJU9dipCZwu2UV0vCsaFbNJEkLxnOfEMdiOtPqkJbVeiG2YRLiCN BnY5wlDx6K2Iru6BfCm0EjH7TgJL8VQJQsmA/9p1l0pbgdBp4AqKx5yJWJmT6UZK4hj1 uFEujr1OWIqau5XkTvAc0hlr61G8YQBtGUGLGozIez3gypaK1cqAy2k6kZaIMMBgYt0s yL1sjewavdoh28ahuI54MLDeTQGxZagPkgHpzF2acyyalkJgOVSn2b1KPnVkhASZO+0M uciA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=buJA7JBdtnLfaYWGzMfKmcyZstmCAza5dcaiN9jkd8o=; b=HhqVwIHlqFBI4mOknTTvyYqoN0/SbYtsL4UE4IFz9t/zkdMas2TZXEYjBQnfdOT5+k vjxr1mcoiawhrnBG1xKCC/mPdhqJI3kFLYGVuuG6oyGofV81gLNs/SL/ORRAL9coPLgp 85+dHgR4XYPygpg86aNWf2CXgBYmC//R0CIYoU00nLF3+/EG0xUNUJ2qCfVUbMCVUwjZ JmFkldxdjtQPiH0SR+xHVMe/ASzT1n0lB0Ds6HGh7HxfBNoKV7EI1WqvdMlk8Qen7AnV 83QKFy+5heNQ3QVT3KFoefUd1DXTFFyXqksfoMHIm0dIQWvnkOU6oNiPO0QKhp0cP9T6 bnBg== X-Gm-Message-State: AOAM530Pphh0+IT7ObctBfTy11J0ohZtOY/9mmMB/nwXPI2RGLeo6FOn bBD/f1PpYHmKrk4s4cOaQIE= X-Google-Smtp-Source: ABdhPJxUH6HG/jKtvX1dbx4scwDTPSup7lGW2Jg3wBucA12nkisFA49VbQ4C21MmFHSi0oZJQ2E3wQ== X-Received: by 2002:a05:6402:13ce:: with SMTP id a14mr15924609edx.365.1614597518895; Mon, 01 Mar 2021 03:18:38 -0800 (PST) Received: from localhost.localdomain ([188.25.217.13]) by smtp.gmail.com with ESMTPSA id i13sm13586491ejj.2.2021.03.01.03.18.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 01 Mar 2021 03:18:38 -0800 (PST) From: Vladimir Oltean To: "David S . Miller" , Jakub Kicinski , netdev@vger.kernel.org Cc: Michael Walle , Claudiu Manoil , Alexandru Marginean , Andrew Lunn , Vladimir Oltean , =?utf-8?q?Markus_Bl=C3=B6chl?= Subject: [PATCH v3 net 5/8] net: enetc: don't disable VLAN filtering in IFF_PROMISC mode Date: Mon, 1 Mar 2021 13:18:15 +0200 Message-Id: <20210301111818.2081582-6-olteanv@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210301111818.2081582-1-olteanv@gmail.com> References: <20210301111818.2081582-1-olteanv@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Vladimir Oltean Quoting from the blamed commit: In promiscuous mode, it is more intuitive that all traffic is received, including VLAN tagged traffic. It appears that it is necessary to set the flag in PSIPVMR for that to be the case, so VLAN promiscuous mode is also temporarily enabled. On exit from promiscuous mode, the setting made by ethtool is restored. Intuitive or not, there isn't any definition issued by a standards body which says that promiscuity has anything to do with VLAN filtering - it only has to do with accepting packets regardless of destination MAC address. In fact people are already trying to use this misunderstanding/bug of the enetc driver as a justification to transform promiscuity into something it never was about: accepting every packet (maybe that would be the "rx-all" netdev feature?): https://lore.kernel.org/netdev/20201110153958.ci5ekor3o2ekg3ky@ipetronik.com/ This is relevant because there are use cases in the kernel (such as tc-flower rules with the protocol 802.1Q and a vlan_id key) which do not (yet) use the vlan_vid_add API to be compatible with VLAN-filtering NICs such as enetc, so for those, disabling rx-vlan-filter is currently the only right solution to make these setups work: https://lore.kernel.org/netdev/CA+h21hoxwRdhq4y+w8Kwgm74d4cA0xLeiHTrmT-VpSaM7obhkg@mail.gmail.com/ The blamed patch has unintentionally introduced one more way for this to work, which is to enable IFF_PROMISC, however this is non-portable because port promiscuity is not meant to disable VLAN filtering. Therefore, it could invite people to write broken scripts for enetc, and then wonder why they are broken when migrating to other drivers that don't handle promiscuity in the same way. Fixes: 7070eea5e95a ("enetc: permit configuration of rx-vlan-filter with ethtool") Cc: Markus Blöchl Signed-off-by: Vladimir Oltean --- Changes in v3: Added one more paragraph in the commit message. Changes in v2: None. drivers/net/ethernet/freescale/enetc/enetc_pf.c | 5 ----- 1 file changed, 5 deletions(-) diff --git a/drivers/net/ethernet/freescale/enetc/enetc_pf.c b/drivers/net/ethernet/freescale/enetc/enetc_pf.c index 62ba4bf56f0d..49681a0566ed 100644 --- a/drivers/net/ethernet/freescale/enetc/enetc_pf.c +++ b/drivers/net/ethernet/freescale/enetc/enetc_pf.c @@ -190,7 +190,6 @@ static void enetc_pf_set_rx_mode(struct net_device *ndev) { struct enetc_ndev_priv *priv = netdev_priv(ndev); struct enetc_pf *pf = enetc_si_priv(priv->si); - char vlan_promisc_simap = pf->vlan_promisc_simap; struct enetc_hw *hw = &priv->si->hw; bool uprom = false, mprom = false; struct enetc_mac_filter *filter; @@ -203,16 +202,12 @@ static void enetc_pf_set_rx_mode(struct net_device *ndev) psipmr = ENETC_PSIPMR_SET_UP(0) | ENETC_PSIPMR_SET_MP(0); uprom = true; mprom = true; - /* Enable VLAN promiscuous mode for SI0 (PF) */ - vlan_promisc_simap |= BIT(0); } else if (ndev->flags & IFF_ALLMULTI) { /* enable multi cast promisc mode for SI0 (PF) */ psipmr = ENETC_PSIPMR_SET_MP(0); mprom = true; } - enetc_set_vlan_promisc(&pf->si->hw, vlan_promisc_simap); - /* first 2 filter entries belong to PF */ if (!uprom) { /* Update unicast filters */ From patchwork Mon Mar 1 11:18:16 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vladimir Oltean X-Patchwork-Id: 390775 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 14CCEC433DB for ; Mon, 1 Mar 2021 11:19:20 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id DA66E64DE0 for ; Mon, 1 Mar 2021 11:19:19 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232122AbhCALTS (ORCPT ); Mon, 1 Mar 2021 06:19:18 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54788 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232272AbhCALTA (ORCPT ); Mon, 1 Mar 2021 06:19:00 -0500 Received: from mail-ed1-x534.google.com (mail-ed1-x534.google.com [IPv6:2a00:1450:4864:20::534]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 14648C061356 for ; Mon, 1 Mar 2021 03:18:41 -0800 (PST) Received: by mail-ed1-x534.google.com with SMTP id d13so15116026edp.4 for ; Mon, 01 Mar 2021 03:18:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=gG4fa3Fc6yf+mOmtYOYZ0TJQvAaAscCQQkSVOKBJq9k=; b=g8xhFFRT9wNeRgEcHAIqXZnPZQK9DmUzqHlLY7L0neSsBeQVdi3j1vaEVNER/KjV59 mg0O7aJYNVgbr0aCPbIM3jDutqQHrRgm/sgmWvGly5zw6NvsKIkpOuu1EZvgNOKfIwv7 Iu9UCQ/CNXe6apYzrLYne6SFCzQ5GoRwyu4GWu7+D1FUA7VzhAvjVsOe1EEIcC0jR6Kp 9Y3T2/rT2/HQ/lXTcJVJDZ+85VDMS1UdOA+vthy8OraGU2jqCF+CdBeExGBcC1Cs+ptQ 54YmlSCbhaUrXQffFcvROgSWmvKZNbu6TellauvKytv9TZ/BnxufJIXNotUab6UfLVIi ADFg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=gG4fa3Fc6yf+mOmtYOYZ0TJQvAaAscCQQkSVOKBJq9k=; b=l4EwKPyhiJcCfA0TfsmQ/F9dRCTHLBWjJYB19MCR37WrfBSnbaV3o51IQo+HzGVS3B 2PyCwBZNA7g6Bnls7yzAiiiUohpfATQ4OAH8acg797UPMwYb4vfScOdvaOVPaPWzYJPT DA4yHNJakLOdqJ+hJA0m0reVTzFIybi2fVkswJtFiXl91pM5R8loMfgCBhdjZupHcjHa lCmOQo3MZMRJO7hVOwaLuotErKytSUAr4E9aJZdRDJ1XGAGHaa1LFuUu0TfBoNuPnMU6 GR/osHGAqgADXJcG7wv52cV3xMMuR0SZpB6LPpxf6YLrYxYG61LDlwHAuJnokOGmz90Y QD/Q== X-Gm-Message-State: AOAM532FzADjLAV2wzhUwBAG/Umms2Csi7H2gA5egy9he1h2JcAqvz97 nVsGC1O9WzMU/A0ziw/vcgg= X-Google-Smtp-Source: ABdhPJz93xzBBydhk/Fh+7J/+hV2/NfH1f/2UTtWavhjJSubtFrXBEWkoxlButT1uxsMUepJjyTRhg== X-Received: by 2002:aa7:ca57:: with SMTP id j23mr16092611edt.293.1614597519886; Mon, 01 Mar 2021 03:18:39 -0800 (PST) Received: from localhost.localdomain ([188.25.217.13]) by smtp.gmail.com with ESMTPSA id i13sm13586491ejj.2.2021.03.01.03.18.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 01 Mar 2021 03:18:39 -0800 (PST) From: Vladimir Oltean To: "David S . Miller" , Jakub Kicinski , netdev@vger.kernel.org Cc: Michael Walle , Claudiu Manoil , Alexandru Marginean , Andrew Lunn , Vladimir Oltean , Florian Fainelli , Russell King Subject: [PATCH v3 net 6/8] net: enetc: force the RGMII speed and duplex instead of operating in inband mode Date: Mon, 1 Mar 2021 13:18:16 +0200 Message-Id: <20210301111818.2081582-7-olteanv@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210301111818.2081582-1-olteanv@gmail.com> References: <20210301111818.2081582-1-olteanv@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Vladimir Oltean The ENETC port 0 MAC supports in-band status signaling coming from a PHY when operating in RGMII mode, and this feature is enabled by default. It has been reported that RGMII is broken in fixed-link, and that is not surprising considering the fact that no PHY is attached to the MAC in that case, but a switch. This brings us to the topic of the patch: the enetc driver should have not enabled the optional in-band status signaling for RGMII unconditionally, but should have forced the speed and duplex to what was resolved by phylink. Note that phylink does not accept the RGMII modes as valid for in-band signaling, and these operate a bit differently than 1000base-x and SGMII (notably there is no clause 37 state machine so no ACK required from the MAC, instead the PHY sends extra code words on RXD[3:0] whenever it is not transmitting something else, so it should be safe to leave a PHY with this option unconditionally enabled even if we ignore it). The spec talks about this here: https://e2e.ti.com/cfs-file/__key/communityserver-discussions-components-files/138/RGMIIv1_5F00_3.pdf Fixes: 71b77a7a27a3 ("enetc: Migrate to PHYLINK and PCS_LYNX") Cc: Florian Fainelli Cc: Andrew Lunn Cc: Russell King Signed-off-by: Vladimir Oltean Acked-by: Russell King --- Changes in v3: Removed the extra "mode" argument passed to enetc_mac_config which was a leftover from v1 (we no longer look at it). Changes in v2: - Don't write to the MAC if speed and duplex did not change. - Don't update the speed with the MAC enabled. - Remove the logic for enabling in-band signaling in enetc_mac_config. .../net/ethernet/freescale/enetc/enetc_hw.h | 13 +++-- .../net/ethernet/freescale/enetc/enetc_pf.c | 53 ++++++++++++++++--- 2 files changed, 56 insertions(+), 10 deletions(-) diff --git a/drivers/net/ethernet/freescale/enetc/enetc_hw.h b/drivers/net/ethernet/freescale/enetc/enetc_hw.h index a62604a1e54e..de0d20b0f489 100644 --- a/drivers/net/ethernet/freescale/enetc/enetc_hw.h +++ b/drivers/net/ethernet/freescale/enetc/enetc_hw.h @@ -238,10 +238,17 @@ enum enetc_bdr_type {TX, RX}; #define ENETC_PM_IMDIO_BASE 0x8030 #define ENETC_PM0_IF_MODE 0x8300 -#define ENETC_PMO_IFM_RG BIT(2) +#define ENETC_PM0_IFM_RG BIT(2) #define ENETC_PM0_IFM_RLP (BIT(5) | BIT(11)) -#define ENETC_PM0_IFM_RGAUTO (BIT(15) | ENETC_PMO_IFM_RG | BIT(1)) -#define ENETC_PM0_IFM_XGMII BIT(12) +#define ENETC_PM0_IFM_EN_AUTO BIT(15) +#define ENETC_PM0_IFM_SSP_MASK GENMASK(14, 13) +#define ENETC_PM0_IFM_SSP_1000 (2 << 13) +#define ENETC_PM0_IFM_SSP_100 (0 << 13) +#define ENETC_PM0_IFM_SSP_10 (1 << 13) +#define ENETC_PM0_IFM_FULL_DPX BIT(12) +#define ENETC_PM0_IFM_IFMODE_MASK GENMASK(1, 0) +#define ENETC_PM0_IFM_IFMODE_XGMII 0 +#define ENETC_PM0_IFM_IFMODE_GMII 2 #define ENETC_PSIDCAPR 0x1b08 #define ENETC_PSIDCAPR_MSK GENMASK(15, 0) #define ENETC_PSFCAPR 0x1b18 diff --git a/drivers/net/ethernet/freescale/enetc/enetc_pf.c b/drivers/net/ethernet/freescale/enetc/enetc_pf.c index 49681a0566ed..ca02f033bea2 100644 --- a/drivers/net/ethernet/freescale/enetc/enetc_pf.c +++ b/drivers/net/ethernet/freescale/enetc/enetc_pf.c @@ -315,7 +315,7 @@ static void enetc_set_loopback(struct net_device *ndev, bool en) u32 reg; reg = enetc_port_rd(hw, ENETC_PM0_IF_MODE); - if (reg & ENETC_PMO_IFM_RG) { + if (reg & ENETC_PM0_IFM_RG) { /* RGMII mode */ reg = (reg & ~ENETC_PM0_IFM_RLP) | (en ? ENETC_PM0_IFM_RLP : 0); @@ -494,13 +494,20 @@ static void enetc_configure_port_mac(struct enetc_hw *hw) static void enetc_mac_config(struct enetc_hw *hw, phy_interface_t phy_mode) { - /* set auto-speed for RGMII */ - if (enetc_port_rd(hw, ENETC_PM0_IF_MODE) & ENETC_PMO_IFM_RG || - phy_interface_mode_is_rgmii(phy_mode)) - enetc_port_wr(hw, ENETC_PM0_IF_MODE, ENETC_PM0_IFM_RGAUTO); + u32 val; + + if (phy_interface_mode_is_rgmii(phy_mode)) { + val = enetc_port_rd(hw, ENETC_PM0_IF_MODE); + val &= ~ENETC_PM0_IFM_EN_AUTO; + val &= ENETC_PM0_IFM_IFMODE_MASK; + val |= ENETC_PM0_IFM_IFMODE_GMII | ENETC_PM0_IFM_RG; + enetc_port_wr(hw, ENETC_PM0_IF_MODE, val); + } - if (phy_mode == PHY_INTERFACE_MODE_USXGMII) - enetc_port_wr(hw, ENETC_PM0_IF_MODE, ENETC_PM0_IFM_XGMII); + if (phy_mode == PHY_INTERFACE_MODE_USXGMII) { + val = ENETC_PM0_IFM_FULL_DPX | ENETC_PM0_IFM_IFMODE_XGMII; + enetc_port_wr(hw, ENETC_PM0_IF_MODE, val); + } } static void enetc_mac_enable(struct enetc_hw *hw, bool en) @@ -932,6 +939,34 @@ static void enetc_pl_mac_config(struct phylink_config *config, phylink_set_pcs(priv->phylink, &pf->pcs->pcs); } +static void enetc_force_rgmii_mac(struct enetc_hw *hw, int speed, int duplex) +{ + u32 old_val, val; + + old_val = val = enetc_port_rd(hw, ENETC_PM0_IF_MODE); + + if (speed == SPEED_1000) { + val &= ~ENETC_PM0_IFM_SSP_MASK; + val |= ENETC_PM0_IFM_SSP_1000; + } else if (speed == SPEED_100) { + val &= ~ENETC_PM0_IFM_SSP_MASK; + val |= ENETC_PM0_IFM_SSP_100; + } else if (speed == SPEED_10) { + val &= ~ENETC_PM0_IFM_SSP_MASK; + val |= ENETC_PM0_IFM_SSP_10; + } + + if (duplex == DUPLEX_FULL) + val |= ENETC_PM0_IFM_FULL_DPX; + else + val &= ~ENETC_PM0_IFM_FULL_DPX; + + if (val == old_val) + return; + + enetc_port_wr(hw, ENETC_PM0_IF_MODE, val); +} + static void enetc_pl_mac_link_up(struct phylink_config *config, struct phy_device *phy, unsigned int mode, phy_interface_t interface, int speed, @@ -944,6 +979,10 @@ static void enetc_pl_mac_link_up(struct phylink_config *config, if (priv->active_offloads & ENETC_F_QBV) enetc_sched_speed_set(priv, speed); + if (!phylink_autoneg_inband(mode) && + phy_interface_mode_is_rgmii(interface)) + enetc_force_rgmii_mac(&pf->si->hw, speed, duplex); + enetc_mac_enable(&pf->si->hw, true); } From patchwork Mon Mar 1 11:18:17 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vladimir Oltean X-Patchwork-Id: 390772 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D26B8C433E0 for ; Mon, 1 Mar 2021 11:20:58 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 967E864DF5 for ; Mon, 1 Mar 2021 11:20:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232583AbhCALU4 (ORCPT ); Mon, 1 Mar 2021 06:20:56 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54898 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232322AbhCALTY (ORCPT ); Mon, 1 Mar 2021 06:19:24 -0500 Received: from mail-ed1-x52a.google.com (mail-ed1-x52a.google.com [IPv6:2a00:1450:4864:20::52a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0B4C6C06121D for ; Mon, 1 Mar 2021 03:18:42 -0800 (PST) Received: by mail-ed1-x52a.google.com with SMTP id l12so20132363edt.3 for ; Mon, 01 Mar 2021 03:18:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=68Pi3ZfOgUhzARIUi2t3kfkoKLegX+yuIQD0xCOE/lo=; b=NwWOzYOJFr+FpwevzAvEVYW3heEligYM3Vzp9sL0X8Wkio5ReToFf1VdcvWnGzks5O gXUzi/MchT1/hlVUgDHT2EYUCzXaHhUWHPJIi2mOz6r11hRx8Ka9hrxi3fg3UbNeA41b GEQCbFkQdyBEFju5z1Yxc3WvUgiZR5e5uiHeNH3W7saBgaGGodFlq+n/nM7OXoQ9Quym JO7mPDCb+naN9UBrGO+JcdvZJ2bphvjW6Bd2kFOZ0fs4rAra+fOGb261gS/9H7fkm0/X d454Istnryq8oBXfMo3wPsG7Yp4E+nGoeMr6wxo3FGYW8bVmQuF61mZRc4SnOZ/iW8kU oN0Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=68Pi3ZfOgUhzARIUi2t3kfkoKLegX+yuIQD0xCOE/lo=; b=eSJZpLGEnTuo0l/SHsOWSA2owYERH1i7KNBDvnKlFkzNgCfdLInewT1q3ZwXrrRgfx BdLpdKOGB23hNpJJNye9s3IKKE/DZynmxqwLgp214dPF6zExABtnuaNvXuf80tS4+ak3 i/P5K8cGLGv5sK/KNu7CFJkmWVh6vZ3zoveyU7ldLbMsbH6A+bNAPSGnmqkzlooGS3Z4 h0lq5T7VeBrL/2/OcjFTesmQM1UCjjmCT73MjLQuLvpVwGZISQlR5S3bp1JftWGWt88O kBK7OwAPGHPgE0lzNkA2CQlstMAQnNgjEG3FiaykbjoGPLYRwo+4eKiwixdWOer2qKhm 3vbQ== X-Gm-Message-State: AOAM533ayxVIHY3B13IMxDP+mH6d25oPSV+KVT6YEpjOXvAXdFk76O02 PzjTaMzCg/PF6MrfjAzPvGk= X-Google-Smtp-Source: ABdhPJx+PcUtc8Y5FBw2hdAhHJDM/n5f7hddXhbOiPglQyluKhLWOuYhSU1D03GuDgVEpxFhyjgCNQ== X-Received: by 2002:a50:e0c3:: with SMTP id j3mr16401349edl.32.1614597520784; Mon, 01 Mar 2021 03:18:40 -0800 (PST) Received: from localhost.localdomain ([188.25.217.13]) by smtp.gmail.com with ESMTPSA id i13sm13586491ejj.2.2021.03.01.03.18.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 01 Mar 2021 03:18:40 -0800 (PST) From: Vladimir Oltean To: "David S . Miller" , Jakub Kicinski , netdev@vger.kernel.org Cc: Michael Walle , Claudiu Manoil , Alexandru Marginean , Andrew Lunn , Vladimir Oltean Subject: [PATCH v3 net 7/8] net: enetc: remove bogus write to SIRXIDR from enetc_setup_rxbdr Date: Mon, 1 Mar 2021 13:18:17 +0200 Message-Id: <20210301111818.2081582-8-olteanv@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210301111818.2081582-1-olteanv@gmail.com> References: <20210301111818.2081582-1-olteanv@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Vladimir Oltean The Station Interface Receive Interrupt Detect Register (SIRXIDR) contains a 16-bit wide mask of 'interrupt detected' events for each ring associated with a port. Bit i is write-1-to-clean for RX ring i. I have no explanation whatsoever how this line of code came to be inserted in the blamed commit. I checked the downstream versions of that patch and none of them have it. The somewhat comical aspect of it is that we're writing a binary number to the SIRXIDR register, which is derived from enetc_bd_unused(rx_ring). Since the RX rings have 512 buffer descriptors, we end up writing 511 to this register, which is 0x1ff, so we are effectively clearing the 'interrupt detected' event for rings 0-8. This register is not what is used for interrupt handling though - it only provides a summary for the entire SI. The hardware provides one separate Interrupt Detect Register per RX ring, which auto-clears upon read. So there doesn't seem to be any adverse effect caused by this bogus write. There is, however, one reason why this should be handled as a bugfix: next_to_clean _should_ be committed to hardware, just not to that register, and this was obscuring the fact that it wasn't. This is fixed in the next patch, and removing the bogus line now allows the fix patch to be backported beyond that point. Fixes: fd5736bf9f23 ("enetc: Workaround for MDIO register access issue") Signed-off-by: Vladimir Oltean --- Changes in v3: Patch is new. drivers/net/ethernet/freescale/enetc/enetc.c | 1 - 1 file changed, 1 deletion(-) diff --git a/drivers/net/ethernet/freescale/enetc/enetc.c b/drivers/net/ethernet/freescale/enetc/enetc.c index 8ddf0cdc37a5..abb29ee81463 100644 --- a/drivers/net/ethernet/freescale/enetc/enetc.c +++ b/drivers/net/ethernet/freescale/enetc/enetc.c @@ -1212,7 +1212,6 @@ static void enetc_setup_rxbdr(struct enetc_hw *hw, struct enetc_bdr *rx_ring) rx_ring->idr = hw->reg + ENETC_SIRXIDR; enetc_refill_rx_ring(rx_ring, enetc_bd_unused(rx_ring)); - enetc_wr(hw, ENETC_SIRXIDR, rx_ring->next_to_use); /* enable ring */ enetc_rxbdr_wr(hw, idx, ENETC_RBMR, rbmr); From patchwork Mon Mar 1 11:18:18 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vladimir Oltean X-Patchwork-Id: 389022 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id ADA31C433E0 for ; Mon, 1 Mar 2021 11:19:25 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 712C864DE0 for ; Mon, 1 Mar 2021 11:19:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232298AbhCALTW (ORCPT ); Mon, 1 Mar 2021 06:19:22 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54790 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232277AbhCALTA (ORCPT ); Mon, 1 Mar 2021 06:19:00 -0500 Received: from mail-ed1-x536.google.com (mail-ed1-x536.google.com [IPv6:2a00:1450:4864:20::536]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EC1E6C06121E for ; Mon, 1 Mar 2021 03:18:42 -0800 (PST) Received: by mail-ed1-x536.google.com with SMTP id h10so20095506edl.6 for ; Mon, 01 Mar 2021 03:18:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=DGRLyWgVglbf27RrOitRngNDUY5X6eKgzS2ML0xYkf8=; b=T+bJWiD16rKGjxo4VrMwsEVkmljNf4GMop4Cbhx03SQdQ2Nzpey6eFqc384qbPuBWs ztYy0+ZermDvC6ctneCkdd7TGniWzaeAZHxYf/gSWhD0btgIMd1deHVTY65OPi2Fr6PU einN/hZtUKPEhqBwh6ve6pPbL0HMeBwr2ipNN/qdJCa5xPzgTh8bHuBrX4mn/LOpigz+ 02UkLH7lqfHfBeC0VkLip/w9nIifJmR5VHM+5/tfhHmXBzjpAoJykui8YnkMWehPtv1H xJs4UQzd1WKmdCm4RUpc7Iu0k4zyK1DEPmsQN7ovWCkZj8Z3KRv0AZHrUzXhK5+N2vqa 0j+g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=DGRLyWgVglbf27RrOitRngNDUY5X6eKgzS2ML0xYkf8=; b=HKuvrueMK1qBTKWKSvXqQyfGouXtCvNUmGxxWxb1gGbcwpE9WyjioY749mQHQGFrvO 0a2SZ3xzbuSTAA/BlrCBRTRGXUvf6uZPwbIuHfol/1jvpAiW9gaAt/1A6T2W2+BBB0iU e+BYyM5OVNZ4AvKnybcXEE875RtUoz4Mxb8hLNZu1ZYZcKPjFzRMPmlgrjOIoJQOOuls kRhe5+jb0olaJ0HrpXPyf35fBxAnYP3oZijXde6GXvI1WyQWe7r1nGp7u5GJcLvWDSnp AFAgLYX+4EryekDHuUEx+cTOHobVeyXr5OZSK/YNJTrkfFGviY91z6ff4zDtgc/pe/hp Bf6g== X-Gm-Message-State: AOAM533OUUg22tdMScAyc+Vqz+PnTWLYRQIt3eaTi/yt6LArkTZwBA3z L1fqGfOJD8eRghrlnttVugA= X-Google-Smtp-Source: ABdhPJz1f6d3xBvL6BsPlksD7uR7TK8VVtU5P33VYq5NXmELEPcD0wEuSgaVR4cwHtpUBRYXqFyutw== X-Received: by 2002:a05:6402:3049:: with SMTP id bu9mr9549044edb.104.1614597521681; Mon, 01 Mar 2021 03:18:41 -0800 (PST) Received: from localhost.localdomain ([188.25.217.13]) by smtp.gmail.com with ESMTPSA id i13sm13586491ejj.2.2021.03.01.03.18.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 01 Mar 2021 03:18:41 -0800 (PST) From: Vladimir Oltean To: "David S . Miller" , Jakub Kicinski , netdev@vger.kernel.org Cc: Michael Walle , Claudiu Manoil , Alexandru Marginean , Andrew Lunn , Vladimir Oltean Subject: [PATCH v3 net 8/8] net: enetc: keep RX ring consumer index in sync with hardware Date: Mon, 1 Mar 2021 13:18:18 +0200 Message-Id: <20210301111818.2081582-9-olteanv@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210301111818.2081582-1-olteanv@gmail.com> References: <20210301111818.2081582-1-olteanv@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Vladimir Oltean The RX rings have a producer index owned by hardware, where newly received frame buffers are placed, and a consumer index owned by software, where newly allocated buffers are placed, in expectation of hardware being able to place frame data in them. Hardware increments the producer index when a frame is received, however it is not allowed to increment the producer index to match the consumer index (RBCIR) since the ring can hold at most RBLENR[LENGTH]-1 received BDs. Whenever the producer index matches the value of the consumer index, the ring has no unprocessed received frames and all BDs in the ring have been initialized/prepared by software, i.e. hardware owns all BDs in the ring. The code uses the next_to_clean variable to keep track of the producer index, and the next_to_use variable to keep track of the consumer index. The RX rings are seeded from enetc_refill_rx_ring, which is called from two places: 1. initially the ring is seeded until full with enetc_bd_unused(rx_ring), i.e. with 511 buffers. This will make next_to_clean=0 and next_to_use=511: .ndo_open -> enetc_open -> enetc_setup_bdrs -> enetc_setup_rxbdr -> enetc_refill_rx_ring 2. then during the data path processing, it is refilled with 16 buffers at a time: enetc_msix -> napi_schedule -> enetc_poll -> enetc_clean_rx_ring -> enetc_refill_rx_ring There is just one problem: the initial seeding done during .ndo_open updates just the producer index (ENETC_RBPIR) with 0, and the software next_to_clean and next_to_use variables. Notably, it will not update the consumer index to make the hardware aware of the newly added buffers. Wait, what? So how does it work? Well, the reset values of the producer index and of the consumer index of a ring are both zero. As per the description in the second paragraph, it means that the ring is full of buffers waiting for hardware to put frames in them, which by coincidence is almost true, because we have in fact seeded 511 buffers into the ring. But will the hardware attempt to access the 512th entry of the ring, which has an invalid BD in it? Well, no, because in order to do that, it would have to first populate the first 511 entries, and the NAPI enetc_poll will kick in by then. Eventually, after 16 processed slots have become available in the RX ring, enetc_clean_rx_ring will call enetc_refill_rx_ring and then will [ finally ] update the consumer index with the new software next_to_use variable. From now on, the next_to_clean and next_to_use variables are in sync with the producer and consumer ring indices. So the day is saved, right? Well, not quite. Freeing the memory allocated for the rings is done in: enetc_close -> enetc_clear_bdrs -> enetc_clear_rxbdr -> this just disables the ring -> enetc_free_rxtx_rings -> enetc_free_rx_ring -> sets next_to_clean and next_to_use to 0 but again, nothing is committed to the hardware producer and consumer indices (yay!). The assumption is that the ring is disabled, so the indices don't matter anyway, and it's the responsibility of the "open" code path to set those up. .. Except that the "open" code path does not set those up properly. While initially, things almost work, during subsequent enetc_close -> enetc_open sequences, we have problems. To be precise, the enetc_open that is subsequent to enetc_close will again refill the ring with 511 entries, but it will leave the consumer index untouched. Untouched means, of course, equal to the value it had before disabling the ring and draining the old buffers in enetc_close. But as mentioned, enetc_setup_rxbdr will at least update the producer index though, through this line of code: enetc_rxbdr_wr(hw, idx, ENETC_RBPIR, 0); so at this stage we'll have: next_to_clean=0 (in hardware 0) next_to_use=511 (in hardware we'll have the refill index prior to enetc_close) Again, the next_to_clean and producer index are in sync and set to correct values, so the driver manages to limp on. Eventually, 16 ring entries will be consumed by enetc_poll, and the savior enetc_clean_rx_ring will come and call enetc_refill_rx_ring, and then update the hardware consumer ring based upon the new next_to_use. So.. it works? Well, by coincidence, it almost does, but there's a circumstance where enetc_clean_rx_ring won't be there to save us. If the previous value of the consumer index was 15, there's a problem, because the NAPI poll sequence will only issue a refill when 16 or more buffers have been consumed. It's easiest to illustrate this with an example: ip link set eno0 up ip addr add 192.168.100.1/24 dev eno0 ping 192.168.100.1 -c 20 # ping this port from another board ip link set eno0 down ip link set eno0 up ping 192.168.100.1 -c 20 # ping it again from the same other board One by one: 1. ip link set eno0 up -> calls enetc_setup_rxbdr: -> calls enetc_refill_rx_ring(511 buffers) -> next_to_clean=0 (in hw 0) -> next_to_use=511 (in hw 0) 2. ping 192.168.100.1 -c 20 # ping this port from another board enetc_clean_rx_ring: rx_frm_cnt=1 cleaned_cnt=1 next_to_clean 0 (in hw 1) next_to_use 511 (in hw 0) enetc_clean_rx_ring: rx_frm_cnt=1 cleaned_cnt=2 next_to_clean 1 (in hw 2) next_to_use 511 (in hw 0) enetc_clean_rx_ring: rx_frm_cnt=1 cleaned_cnt=3 next_to_clean 2 (in hw 3) next_to_use 511 (in hw 0) enetc_clean_rx_ring: rx_frm_cnt=1 cleaned_cnt=4 next_to_clean 3 (in hw 4) next_to_use 511 (in hw 0) enetc_clean_rx_ring: rx_frm_cnt=1 cleaned_cnt=5 next_to_clean 4 (in hw 5) next_to_use 511 (in hw 0) enetc_clean_rx_ring: rx_frm_cnt=1 cleaned_cnt=6 next_to_clean 5 (in hw 6) next_to_use 511 (in hw 0) enetc_clean_rx_ring: rx_frm_cnt=1 cleaned_cnt=7 next_to_clean 6 (in hw 7) next_to_use 511 (in hw 0) enetc_clean_rx_ring: rx_frm_cnt=1 cleaned_cnt=8 next_to_clean 7 (in hw 8) next_to_use 511 (in hw 0) enetc_clean_rx_ring: rx_frm_cnt=1 cleaned_cnt=9 next_to_clean 8 (in hw 9) next_to_use 511 (in hw 0) enetc_clean_rx_ring: rx_frm_cnt=1 cleaned_cnt=10 next_to_clean 9 (in hw 10) next_to_use 511 (in hw 0) enetc_clean_rx_ring: rx_frm_cnt=1 cleaned_cnt=11 next_to_clean 10 (in hw 11) next_to_use 511 (in hw 0) enetc_clean_rx_ring: rx_frm_cnt=1 cleaned_cnt=12 next_to_clean 11 (in hw 12) next_to_use 511 (in hw 0) enetc_clean_rx_ring: rx_frm_cnt=1 cleaned_cnt=13 next_to_clean 12 (in hw 13) next_to_use 511 (in hw 0) enetc_clean_rx_ring: rx_frm_cnt=1 cleaned_cnt=14 next_to_clean 13 (in hw 14) next_to_use 511 (in hw 0) enetc_clean_rx_ring: rx_frm_cnt=1 cleaned_cnt=15 next_to_clean 14 (in hw 15) next_to_use 511 (in hw 0) enetc_clean_rx_ring: enetc_refill_rx_ring(16) increments next_to_use by 16 (mod 512) and writes it to hw enetc_clean_rx_ring: rx_frm_cnt=1 cleaned_cnt=0 next_to_clean 15 (in hw 16) next_to_use 15 (in hw 15) enetc_clean_rx_ring: rx_frm_cnt=1 cleaned_cnt=1 next_to_clean 16 (in hw 17) next_to_use 15 (in hw 15) enetc_clean_rx_ring: rx_frm_cnt=1 cleaned_cnt=2 next_to_clean 17 (in hw 18) next_to_use 15 (in hw 15) enetc_clean_rx_ring: rx_frm_cnt=1 cleaned_cnt=3 next_to_clean 18 (in hw 19) next_to_use 15 (in hw 15) enetc_clean_rx_ring: rx_frm_cnt=1 cleaned_cnt=4 next_to_clean 19 (in hw 20) next_to_use 15 (in hw 15) enetc_clean_rx_ring: rx_frm_cnt=1 cleaned_cnt=5 next_to_clean 20 (in hw 21) next_to_use 15 (in hw 15) enetc_clean_rx_ring: rx_frm_cnt=1 cleaned_cnt=6 next_to_clean 21 (in hw 22) next_to_use 15 (in hw 15) 20 packets transmitted, 20 packets received, 0% packet loss 3. ip link set eno0 down enetc_free_rx_ring: next_to_clean 0 (in hw 22), next_to_use 0 (in hw 15) 4. ip link set eno0 up -> calls enetc_setup_rxbdr: -> calls enetc_refill_rx_ring(511 buffers) -> next_to_clean=0 (in hw 0) -> next_to_use=511 (in hw 15) 5. ping 192.168.100.1 -c 20 # ping it again from the same other board enetc_clean_rx_ring: rx_frm_cnt=1 cleaned_cnt=1 next_to_clean 0 (in hw 1) next_to_use 511 (in hw 15) enetc_clean_rx_ring: rx_frm_cnt=1 cleaned_cnt=2 next_to_clean 1 (in hw 2) next_to_use 511 (in hw 15) enetc_clean_rx_ring: rx_frm_cnt=1 cleaned_cnt=3 next_to_clean 2 (in hw 3) next_to_use 511 (in hw 15) enetc_clean_rx_ring: rx_frm_cnt=1 cleaned_cnt=4 next_to_clean 3 (in hw 4) next_to_use 511 (in hw 15) enetc_clean_rx_ring: rx_frm_cnt=1 cleaned_cnt=5 next_to_clean 4 (in hw 5) next_to_use 511 (in hw 15) enetc_clean_rx_ring: rx_frm_cnt=1 cleaned_cnt=6 next_to_clean 5 (in hw 6) next_to_use 511 (in hw 15) enetc_clean_rx_ring: rx_frm_cnt=1 cleaned_cnt=7 next_to_clean 6 (in hw 7) next_to_use 511 (in hw 15) enetc_clean_rx_ring: rx_frm_cnt=1 cleaned_cnt=8 next_to_clean 7 (in hw 8) next_to_use 511 (in hw 15) enetc_clean_rx_ring: rx_frm_cnt=1 cleaned_cnt=9 next_to_clean 8 (in hw 9) next_to_use 511 (in hw 15) enetc_clean_rx_ring: rx_frm_cnt=1 cleaned_cnt=10 next_to_clean 9 (in hw 10) next_to_use 511 (in hw 15) enetc_clean_rx_ring: rx_frm_cnt=1 cleaned_cnt=11 next_to_clean 10 (in hw 11) next_to_use 511 (in hw 15) enetc_clean_rx_ring: rx_frm_cnt=1 cleaned_cnt=12 next_to_clean 11 (in hw 12) next_to_use 511 (in hw 15) enetc_clean_rx_ring: rx_frm_cnt=1 cleaned_cnt=13 next_to_clean 12 (in hw 13) next_to_use 511 (in hw 15) enetc_clean_rx_ring: rx_frm_cnt=1 cleaned_cnt=14 next_to_clean 13 (in hw 14) next_to_use 511 (in hw 15) 20 packets transmitted, 12 packets received, 40% packet loss And there it dies. No enetc_refill_rx_ring (because cleaned_cnt must be equal to 15 for that to happen), no nothing. The hardware enters the condition where the producer (14) + 1 is equal to the consumer (15) index, which makes it believe it has no more free buffers to put packets in, so it starts discarding them: ip netns exec ns0 ethtool -S eno0 | grep -v ': 0' NIC statistics: Rx ring 0 discarded frames: 8 Summarized, if the interface receives between 16 and 32 (mod 512) frames and then there is a link flap, then the port will eventually die with no way to recover. If it receives less than 16 (mod 512) frames, then the initial NAPI poll [ before the link flap ] will not update the consumer index in hardware (it will remain zero) which will be ok when the buffers are later reinitialized. If more than 32 (mod 512) frames are received, the initial NAPI poll has the chance to refill the ring twice, updating the consumer index to at least 32. So after the link flap, the consumer index is still wrong, but the post-flap NAPI poll gets a chance to refill the ring once (because it passes through cleaned_cnt=15) and makes the consumer index be again back in sync with next_to_use. The solution to this problem is actually simple, we just need to write next_to_use into the hardware consumer index at enetc_open time, which always brings it back in sync after an initial buffer seeding process. The simpler thing would be to put the write to the consumer index into enetc_refill_rx_ring directly, but there are issues with the MDIO locking: in the NAPI poll code we have the enetc_lock_mdio() taken from top-level and we use the unlocked enetc_wr_reg_hot, whereas in enetc_open, the enetc_lock_mdio() is not taken at the top level, but instead by each individual enetc_wr_reg, so we are forced to put an additional enetc_wr_reg in enetc_setup_rxbdr. Better organization of the code is left as a refactoring exercise. Fixes: d4fd0404c1c9 ("enetc: Introduce basic PF and VF ENETC ethernet drivers") Signed-off-by: Vladimir Oltean --- Changes in v3: Patch is new. drivers/net/ethernet/freescale/enetc/enetc.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/drivers/net/ethernet/freescale/enetc/enetc.c b/drivers/net/ethernet/freescale/enetc/enetc.c index abb29ee81463..30d7d4e83900 100644 --- a/drivers/net/ethernet/freescale/enetc/enetc.c +++ b/drivers/net/ethernet/freescale/enetc/enetc.c @@ -1212,6 +1212,8 @@ static void enetc_setup_rxbdr(struct enetc_hw *hw, struct enetc_bdr *rx_ring) rx_ring->idr = hw->reg + ENETC_SIRXIDR; enetc_refill_rx_ring(rx_ring, enetc_bd_unused(rx_ring)); + /* update ENETC's consumer index */ + enetc_rxbdr_wr(hw, idx, ENETC_RBCIR, rx_ring->next_to_use); /* enable ring */ enetc_rxbdr_wr(hw, idx, ENETC_RBMR, rbmr);