From patchwork Fri Oct 2 00:07:10 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Honnappa Nagarahalli X-Patchwork-Id: 313961 Delivered-To: patch@linaro.org Received: by 2002:a05:6e02:1081:0:0:0:0 with SMTP id r1csp1004288ilj; Thu, 1 Oct 2020 17:07:44 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwbRRJ3gHb7pTDZonehIOtk1tu39FopjFBt6aqylbk0jwXEpXI4DBH3lx879rT6nypYIV+D X-Received: by 2002:a05:6402:7c8:: with SMTP id u8mr4681436edy.153.1601597264673; Thu, 01 Oct 2020 17:07:44 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1601597264; cv=none; d=google.com; s=arc-20160816; b=Uz7Ecy+X4Y97dBX8gv/cBgP5pWqRL4ctsDmlmKd9mqz2GQwUXtnfqGUT3/xOfVQ2Xo Qy4/1f/KOybEeWE0q2v5GXPECQsWWivH5H9OSbFPjCJ9HnaeX25gMTScriQ8UFM9PiYo TAO6ZNyrUt62D61TakN0vgEoGEbEIrvonOvaXxMrHPLcCZ3jm1AB04jzj+cESKU4Gw8P PNWuWLpP3EDfk3LCyPDzv9z1PhR1qZChWjYvd2suwRSGHJVSwnpOu5A0FldHvfp8hqWp NbBPZZEdsY1vU92m1+sdvmjnG2YcBHav7NRuOxcksHJiR0G9AiiNimG8hPV2gClKW5wX uOgg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:message-id:date:cc:to :from; bh=nn7iCf9PP9lcny7RCduKCi7N50aye+EHZM0DME3L9JI=; b=bJr42AGqUG9n4KwZ3FrlKYmE6aewcvmx8bXFlklPcDcsl4aeItfaIRicGAd6f5qhXJ JGj3REsUxRoODE5OyrCulCjcHR/Oi221y4xEoVL9BhMN/lQhFCPpjk50WfXzVqJuXn15 76ziLWFZpBlw4S0fSu0fdXviXljrFp2CYfL5vxvhlnM/+tMVVha40DTzFqNqz+LFV70Y G96uRN+l0IuKajtnVbFEmCxMaxeAFhUYQZHWjbuLd5+FIPqVk5gBLvqIap8L82rxirv4 T18AhToDroKZjbE1MdBDHr+HvN9WTZ6y9a/Qb9aTBBnVbo0mkU1x0+lRLkmnJygUzOVI yALg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of dev-bounces@dpdk.org designates 92.243.14.124 as permitted sender) smtp.mailfrom=dev-bounces@dpdk.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Return-Path: Received: from dpdk.org (dpdk.org. [92.243.14.124]) by mx.google.com with ESMTP id s26si4539019edw.32.2020.10.01.17.07.43; Thu, 01 Oct 2020 17:07:44 -0700 (PDT) Received-SPF: pass (google.com: domain of dev-bounces@dpdk.org designates 92.243.14.124 as permitted sender) client-ip=92.243.14.124; Authentication-Results: mx.google.com; spf=pass (google.com: domain of dev-bounces@dpdk.org designates 92.243.14.124 as permitted sender) smtp.mailfrom=dev-bounces@dpdk.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 508291D619; Fri, 2 Oct 2020 02:07:38 +0200 (CEST) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by dpdk.org (Postfix) with ESMTP id 0E09A1D614 for ; Fri, 2 Oct 2020 02:07:37 +0200 (CEST) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 68F2B1042; Thu, 1 Oct 2020 17:07:35 -0700 (PDT) Received: from qc2400f-1.austin.arm.com (qc2400f-1.austin.arm.com [10.118.12.27]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 55FDA3F6CF; Thu, 1 Oct 2020 17:07:35 -0700 (PDT) From: Honnappa Nagarahalli To: dev@dpdk.org, honnappa.nagarahalli@arm.com, phil.yang@arm.com, thomas@monjalon.net, arybchenko@solarflare.com, ferruh.yigit@intel.com Cc: abhinandan.gujjar@intel.com, nd@arm.com Date: Thu, 1 Oct 2020 19:07:10 -0500 Message-Id: <20201002000711.41511-1-honnappa.nagarahalli@arm.com> X-Mailer: git-send-email 2.17.1 Subject: [dpdk-dev] [PATCH 1/2] lib/ethdev: replace full barrier with relaxed barrier X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" From: Phil Yang While registering the call back functions full write barrier can be replaced with one-way write barrier. Signed-off-by: Phil Yang Signed-off-by: Honnappa Nagarahalli Reviewed-by: Ruifeng Wang --- lib/librte_ethdev/rte_ethdev.c | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) -- 2.17.1 Acked-by: Konstantin Ananyev diff --git a/lib/librte_ethdev/rte_ethdev.c b/lib/librte_ethdev/rte_ethdev.c index 7858ad5f1..59a41c07f 100644 --- a/lib/librte_ethdev/rte_ethdev.c +++ b/lib/librte_ethdev/rte_ethdev.c @@ -26,7 +26,6 @@ #include #include #include -#include #include #include #include @@ -4527,8 +4526,12 @@ rte_eth_add_first_rx_callback(uint16_t port_id, uint16_t queue_id, rte_spinlock_lock(&rte_eth_rx_cb_lock); /* Add the callbacks at first position */ cb->next = rte_eth_devices[port_id].post_rx_burst_cbs[queue_id]; - rte_smp_wmb(); - rte_eth_devices[port_id].post_rx_burst_cbs[queue_id] = cb; + /* Stores to cb->fn, cb->param and cb->next should complete before + * cb is visible to data plane threads. + */ + __atomic_store_n( + &rte_eth_devices[port_id].post_rx_burst_cbs[queue_id], + cb, __ATOMIC_RELEASE); rte_spinlock_unlock(&rte_eth_rx_cb_lock); return cb; From patchwork Fri Oct 2 00:07:11 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Honnappa Nagarahalli X-Patchwork-Id: 313962 Delivered-To: patch@linaro.org Received: by 2002:a05:6e02:1081:0:0:0:0 with SMTP id r1csp1004452ilj; Thu, 1 Oct 2020 17:08:01 -0700 (PDT) X-Google-Smtp-Source: ABdhPJyPXYFzEiHJOnPvgIUAwQXZY9pMv5l+QSKScezugg2aWEHfGDz9XhjPzgFuFcmWqIDcrGVT X-Received: by 2002:a17:906:9701:: with SMTP id k1mr11358577ejx.0.1601597281319; Thu, 01 Oct 2020 17:08:01 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1601597281; cv=none; d=google.com; s=arc-20160816; b=l8gMvbOA3DKc06T96OW2KIxVLLucUuDr8zTJEiQsxi8qX8nfiuuxjAWg6QS722HBiV AmOTGB0gFvoDQH1gJ2QQtA/0vcYlnm+gi6rkWgupkqDepo2Rclhz1O82W5ZhjDWrDssb 4DTQshorImFrYphZUJ2aPaUyHRsPETTGud99j4Fxr2yUSVP7+wP64COcncCPT3hN0n6b GHUXQz1ua0zSr3aJTgwpfA88kDD2M+Y9foaH/78Q7BX1N77rDqvizI0p1E+AiU4DARan zyO1vPGmItfjrrI+9KyDD790Gk0/eQRWnNra7pMbGwSN5AjGVS8LmwOENxOzdYPPfdZs mzFw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:references:in-reply-to :message-id:date:cc:to:from; bh=aZ+/kASSdwKQbyeyDDxAd+uYVBzUBIHBeFr1AWr6Y+8=; b=EsWE9ptgN/JOXgP6mG1AKn+7VWFv3q62UwsjThmRjvn6OoomKieQ1k91nKo97q0JcK u7ftuAL/EYMCf2h1JmWdCJ3MDjOlqH0VVut1pgJ941I1GFOI+KVSGBloXf0Zpm8HTy+g /T7akS9NYBGcy6aTeqG9a29IhGZcejIvjRAfYhIQtKCnNW5UtBrS/MOnMEyiKM/jEc/B ANnp4280uonoMYM1L7JHjfbD2teFbvbDg1wWn9obnk4+vuWhtZJ1tbzw5RZoz0lDtfT0 xOI0dQrP5Mf3Yu8V5jy3XqZOlksczaC0O8QC0yChoL24gZj8B52eajjziJzdcN4x8SGX tS+g== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of dev-bounces@dpdk.org designates 92.243.14.124 as permitted sender) smtp.mailfrom=dev-bounces@dpdk.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Return-Path: Received: from dpdk.org (dpdk.org. [92.243.14.124]) by mx.google.com with ESMTP id q23si4469831ejd.363.2020.10.01.17.08.01; Thu, 01 Oct 2020 17:08:01 -0700 (PDT) Received-SPF: pass (google.com: domain of dev-bounces@dpdk.org designates 92.243.14.124 as permitted sender) client-ip=92.243.14.124; Authentication-Results: mx.google.com; spf=pass (google.com: domain of dev-bounces@dpdk.org designates 92.243.14.124 as permitted sender) smtp.mailfrom=dev-bounces@dpdk.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id C7C041D66E; Fri, 2 Oct 2020 02:07:44 +0200 (CEST) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by dpdk.org (Postfix) with ESMTP id 0E6E11D65E; Fri, 2 Oct 2020 02:07:42 +0200 (CEST) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 787E91063; Thu, 1 Oct 2020 17:07:40 -0700 (PDT) Received: from qc2400f-1.austin.arm.com (qc2400f-1.austin.arm.com [10.118.12.27]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 63A5B3F6CF; Thu, 1 Oct 2020 17:07:40 -0700 (PDT) From: Honnappa Nagarahalli To: dev@dpdk.org, honnappa.nagarahalli@arm.com, phil.yang@arm.com, thomas@monjalon.net, arybchenko@solarflare.com, ferruh.yigit@intel.com Cc: abhinandan.gujjar@intel.com, nd@arm.com, bruce.richardson@intel.com, john.mcnamara@intel.com, reshma.pattan@intel.com, stable@dpdk.org Date: Thu, 1 Oct 2020 19:07:11 -0500 Message-Id: <20201002000711.41511-2-honnappa.nagarahalli@arm.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20201002000711.41511-1-honnappa.nagarahalli@arm.com> References: <20201002000711.41511-1-honnappa.nagarahalli@arm.com> Subject: [dpdk-dev] [PATCH 2/2] lib/ethdev: fix memory ordering for call back functions X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Call back functions are registered on the control plane. They are accessed from the data plane. Hence, correct memory orderings should be used to avoid race conditions. Fixes: 4dc294158cac ("ethdev: support optional Rx and Tx callbacks") Fixes: c8231c63ddcb ("ethdev: insert Rx callback as head of list") Cc: bruce.richardson@intel.com Cc: john.mcnamara@intel.com Cc: reshma.pattan@intel.com Cc: stable@dpdk.org Signed-off-by: Honnappa Nagarahalli Reviewed-by: Ola Liljedahl --- lib/librte_ethdev/rte_ethdev.c | 28 +++++++++++++++++++++------ lib/librte_ethdev/rte_ethdev.h | 35 ++++++++++++++++++++++++++-------- 2 files changed, 49 insertions(+), 14 deletions(-) -- 2.17.1 Reviewed-by: Phil Yang Acked-by: Konstantin Ananyev diff --git a/lib/librte_ethdev/rte_ethdev.c b/lib/librte_ethdev/rte_ethdev.c index 59a41c07f..d89fcdc77 100644 --- a/lib/librte_ethdev/rte_ethdev.c +++ b/lib/librte_ethdev/rte_ethdev.c @@ -4486,12 +4486,20 @@ rte_eth_add_rx_callback(uint16_t port_id, uint16_t queue_id, rte_eth_devices[port_id].post_rx_burst_cbs[queue_id]; if (!tail) { - rte_eth_devices[port_id].post_rx_burst_cbs[queue_id] = cb; + /* Stores to cb->fn and cb->param should complete before + * cb is visible to data plane. + */ + __atomic_store_n( + &rte_eth_devices[port_id].post_rx_burst_cbs[queue_id], + cb, __ATOMIC_RELEASE); } else { while (tail->next) tail = tail->next; - tail->next = cb; + /* Stores to cb->fn and cb->param should complete before + * cb is visible to data plane. + */ + __atomic_store_n(&tail->next, cb, __ATOMIC_RELEASE); } rte_spinlock_unlock(&rte_eth_rx_cb_lock); @@ -4576,12 +4584,20 @@ rte_eth_add_tx_callback(uint16_t port_id, uint16_t queue_id, rte_eth_devices[port_id].pre_tx_burst_cbs[queue_id]; if (!tail) { - rte_eth_devices[port_id].pre_tx_burst_cbs[queue_id] = cb; + /* Stores to cb->fn and cb->param should complete before + * cb is visible to data plane. + */ + __atomic_store_n( + &rte_eth_devices[port_id].pre_tx_burst_cbs[queue_id], + cb, __ATOMIC_RELEASE); } else { while (tail->next) tail = tail->next; - tail->next = cb; + /* Stores to cb->fn and cb->param should complete before + * cb is visible to data plane. + */ + __atomic_store_n(&tail->next, cb, __ATOMIC_RELEASE); } rte_spinlock_unlock(&rte_eth_tx_cb_lock); @@ -4612,7 +4628,7 @@ rte_eth_remove_rx_callback(uint16_t port_id, uint16_t queue_id, cb = *prev_cb; if (cb == user_cb) { /* Remove the user cb from the callback list. */ - *prev_cb = cb->next; + __atomic_store_n(prev_cb, cb->next, __ATOMIC_RELAXED); ret = 0; break; } @@ -4646,7 +4662,7 @@ rte_eth_remove_tx_callback(uint16_t port_id, uint16_t queue_id, cb = *prev_cb; if (cb == user_cb) { /* Remove the user cb from the callback list. */ - *prev_cb = cb->next; + __atomic_store_n(prev_cb, cb->next, __ATOMIC_RELAXED); ret = 0; break; } diff --git a/lib/librte_ethdev/rte_ethdev.h b/lib/librte_ethdev/rte_ethdev.h index 70295d7ab..d810e3e38 100644 --- a/lib/librte_ethdev/rte_ethdev.h +++ b/lib/librte_ethdev/rte_ethdev.h @@ -3734,7 +3734,8 @@ struct rte_eth_rxtx_callback; * The callback function * @param user_param * A generic pointer parameter which will be passed to each invocation of the - * callback function on this port and queue. + * callback function on this port and queue. Inter-thread synchronization + * of any user data changes is the responsibility of the user. * * @return * NULL on error. @@ -3763,7 +3764,8 @@ rte_eth_add_rx_callback(uint16_t port_id, uint16_t queue_id, * The callback function * @param user_param * A generic pointer parameter which will be passed to each invocation of the - * callback function on this port and queue. + * callback function on this port and queue. Inter-thread synchronization + * of any user data changes is the responsibility of the user. * * @return * NULL on error. @@ -3791,7 +3793,8 @@ rte_eth_add_first_rx_callback(uint16_t port_id, uint16_t queue_id, * The callback function * @param user_param * A generic pointer parameter which will be passed to each invocation of the - * callback function on this port and queue. + * callback function on this port and queue. Inter-thread synchronization + * of any user data changes is the responsibility of the user. * * @return * NULL on error. @@ -3816,7 +3819,9 @@ rte_eth_add_tx_callback(uint16_t port_id, uint16_t queue_id, * on that queue. * * - After a short delay - where the delay is sufficient to allow any - * in-flight callbacks to complete. + * in-flight callbacks to complete. Alternately, the RCU mechanism can be + * used to detect when data plane threads have ceased referencing the + * callback memory. * * @param port_id * The port identifier of the Ethernet device. @@ -3849,7 +3854,9 @@ int rte_eth_remove_rx_callback(uint16_t port_id, uint16_t queue_id, * on that queue. * * - After a short delay - where the delay is sufficient to allow any - * in-flight callbacks to complete. + * in-flight callbacks to complete. Alternately, the RCU mechanism can be + * used to detect when data plane threads have ceased referencing the + * callback memory. * * @param port_id * The port identifier of the Ethernet device. @@ -4510,10 +4517,16 @@ rte_eth_rx_burst(uint16_t port_id, uint16_t queue_id, rx_pkts, nb_pkts); #ifdef RTE_ETHDEV_RXTX_CALLBACKS - if (unlikely(dev->post_rx_burst_cbs[queue_id] != NULL)) { - struct rte_eth_rxtx_callback *cb = - dev->post_rx_burst_cbs[queue_id]; + /* __ATOMIC_RELEASE memory order was used when the + * call back was inserted into the list. + * Since there is a clear dependency between loading + * cb and cb->fn/cb->next, __ATOMIC_ACQUIRE memory order is + * not required. + */ + struct rte_eth_rxtx_callback *cb = + dev->post_rx_burst_cbs[queue_id]; + if (unlikely(cb != NULL)) { do { nb_rx = cb->fn.rx(port_id, queue_id, rx_pkts, nb_rx, nb_pkts, cb->param); @@ -4775,6 +4788,12 @@ rte_eth_tx_burst(uint16_t port_id, uint16_t queue_id, #endif #ifdef RTE_ETHDEV_RXTX_CALLBACKS + /* __ATOMIC_RELEASE memory order was used when the + * call back was inserted into the list. + * Since there is a clear dependency between loading + * cb and cb->fn/cb->next, __ATOMIC_ACQUIRE memory order is + * not required. + */ struct rte_eth_rxtx_callback *cb = dev->pre_tx_burst_cbs[queue_id]; if (unlikely(cb != NULL)) {