From patchwork Fri Apr 3 18:41:41 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Honnappa Nagarahalli X-Patchwork-Id: 185176 Delivered-To: patch@linaro.org Received: by 2002:a92:de47:0:0:0:0:0 with SMTP id e7csp882541ilr; Fri, 3 Apr 2020 11:42:21 -0700 (PDT) X-Google-Smtp-Source: APiQypLsCxg2NgBYU7SjVPznD4IP2Jnuq7CRA/2wQJrnrzvZONVP3lr5jaeDJZYZIHYM7LOjLoiM X-Received: by 2002:adf:e6c8:: with SMTP id y8mr9985479wrm.279.1585939341863; Fri, 03 Apr 2020 11:42:21 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1585939341; cv=none; d=google.com; s=arc-20160816; b=W7WTf9ZWuj8cr8Kcl1HFGzIRxz9sAkLDKTgu6LFejztb+iAvr+zOkh/uT8tN2Jkc9S xhXnen4i5nXxTpi91LbPAD4u9UNJHbN8+hPZuc2FfHrmTVHwvcH/uA76kNaYU3AYCVKy hOiQBjTSm5cApR3lAx3Z1GyLA/1GuKfxDiWcC35VvJGWLlnFtgLyOSll5PudfJ9tHmtH bW0NRBJPGaiQ5Y2bbu2X9CpEla/Lj2RrmTKNJa5hE9HuwWh48o817OZDl2o1h1tjDEml nCd0rFMzxwHYUEJQSmTTdQo2P/58xAhQUQ86fiXZcATb9DqIDiH4Blel9rnWFe9x0f9I +QzA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:references:in-reply-to :message-id:date:cc:to:from; bh=ZrqSEMp/DAwtZXcEjmBy5lcyb5iGyUMo+qQH90tNDOY=; b=oBPzHVh7jpaoVFKI2EUv3OH5j2zPbZ5IaR0tq9ZjCq5PzsSH5PVzWV5gegJOQ+2mk4 J1eYgY7ylvF+yFmb6Tc96eW4EeIJmG5o+Dcyk3xOV7k2l5UEKEnI6hXGF97LqTXQadms 6IFPxoGob7sJKbRcwa4MYqBgZF0gxTil1o6C8loNI5dxcuAXGJvNzFLBBJZ0paqOMBVB i0HcFOEWm+roadgBo6gNKrRbbeFn9gbrml1mhIpAS4+aYP9RGtmR1+bKDANtk8f5BekJ eTGnc9Wi++abyqJQazFaJVH0mS7V//S80qf6G8JdIVcY8uwC8TcCLtcMUbwTudXtkz6K 4dGg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of dev-bounces@dpdk.org designates 92.243.14.124 as permitted sender) smtp.mailfrom=dev-bounces@dpdk.org Return-Path: Received: from dpdk.org (dpdk.org. [92.243.14.124]) by mx.google.com with ESMTP id b2si8570894wmc.62.2020.04.03.11.42.21; Fri, 03 Apr 2020 11:42:21 -0700 (PDT) Received-SPF: pass (google.com: domain of dev-bounces@dpdk.org designates 92.243.14.124 as permitted sender) client-ip=92.243.14.124; Authentication-Results: mx.google.com; spf=pass (google.com: domain of dev-bounces@dpdk.org designates 92.243.14.124 as permitted sender) smtp.mailfrom=dev-bounces@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id D4CA31C1EB; Fri, 3 Apr 2020 20:42:01 +0200 (CEST) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by dpdk.org (Postfix) with ESMTP id D96FA1C1D4 for ; Fri, 3 Apr 2020 20:41:59 +0200 (CEST) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 636891FB; Fri, 3 Apr 2020 11:41:59 -0700 (PDT) Received: from qc2400f-1.austin.arm.com (qc2400f-1.austin.arm.com [10.118.14.48]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 5192B3F71E; Fri, 3 Apr 2020 11:41:59 -0700 (PDT) From: Honnappa Nagarahalli To: konstantin.ananyev@intel.com, stephen@networkplumber.org, vladimir.medvedkin@intel.com Cc: dev@dpdk.org, honnappa.nagarahalli@arm.com, ruifeng.wang@arm.com, dharmik.thakkar@arm.com, nd@arm.com Date: Fri, 3 Apr 2020 13:41:41 -0500 Message-Id: <20200403184142.7729-4-honnappa.nagarahalli@arm.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200403184142.7729-1-honnappa.nagarahalli@arm.com> References: <20191001062917.35578-1-honnappa.nagarahalli@arm.com> <20200403184142.7729-1-honnappa.nagarahalli@arm.com> Subject: [dpdk-dev] [PATCH v4 3/4] doc/rcu: add RCU integration design details X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" From: Ruifeng Wang Add a section to describe a design to integrate QSBR RCU library with other libraries in DPDK. Signed-off-by: Honnappa Nagarahalli --- doc/guides/prog_guide/rcu_lib.rst | 59 +++++++++++++++++++++++++++++++ 1 file changed, 59 insertions(+) -- 2.17.1 diff --git a/doc/guides/prog_guide/rcu_lib.rst b/doc/guides/prog_guide/rcu_lib.rst index 9b0bf138f..484c7b882 100644 --- a/doc/guides/prog_guide/rcu_lib.rst +++ b/doc/guides/prog_guide/rcu_lib.rst @@ -190,3 +190,62 @@ However, when ``CONFIG_RTE_LIBRTE_RCU_DEBUG`` is enabled, these APIs aid in debugging issues. One can mark the access to shared data structures on the reader side using these APIs. The ``rte_rcu_qsbr_quiescent()`` will check if all the locks are unlocked. + +Resource reclamation framework for DPDK +--------------------------------------- + +Lock-free algorithms place additional burden of resource reclamation on +the application. When a writer deletes an entry from a data structure, the writer: + +#. Has to start the grace period +#. Has to store a reference to the deleted resources in a FIFO +#. Should check if the readers have completed a grace period and free the resources. This can also be done when the writer runs out of free resources. + +There are several APIs provided to help with this process. The writer +can create a FIFO to store the references to deleted resources using ``rte_rcu_qsbr_dq_create()``. +The resources can be enqueued to this FIFO using ``rte_rcu_qsbr_dq_enqueue()``. +If the FIFO is full, ``rte_rcu_qsbr_dq_enqueue`` will reclaim the resources before enqueuing. It will also reclaim resources on regular basis to keep the FIFO from growing too large. If the writer runs out of resources, the writer can call ``rte_rcu_qsbr_dq_reclaim`` API to reclaim resources. ``rte_rcu_qsbr_dq_delete`` is provided to reclaim any remaining resources and free the FIFO while shutting down. + +However, if this resource reclamation process were to be integrated in lock-free data structure libraries, it +hides this complexity from the application and makes it easier for the application to adopt lock-free algorithms. The following paragraphs discuss how the reclamation process can be integrated in DPDK libraries. + +In any DPDK application, the resource reclamation process using QSBR can be split into 4 parts: + +#. Initialization +#. Quiescent State Reporting +#. Reclaiming Resources +#. Shutdown + +The design proposed here assigns different parts of this process to client libraries and applications. The term 'client library' refers to lock-free data structure libraries such at rte_hash, rte_lpm etc. in DPDK or similar libraries outside of DPDK. The term 'application' refers to the packet processing application that makes use of DPDK such as L3 Forwarding example application, OVS, VPP etc.. + +The application has to handle 'Initialization' and 'Quiescent State Reporting'. So, + +* the application has to create the RCU variable and register the reader threads to report their quiescent state. +* the application has to register the same RCU variable with the client library. +* reader threads in the application have to report the quiescent state. This allows for the application to control the length of the critical section/how frequently the application wants to report the quiescent state. + +The client library will handle 'Reclaiming Resources' part of the process. The +client libraries will make use of the writer thread context to execute the memory +reclamation algorithm. So, + +* client library should provide an API to register a RCU variable that it will use. It should call ``rte_rcu_qsbr_dq_create()`` to create the FIFO to store the references to deleted entries. +* client library should use ``rte_rcu_qsbr_dq_enqueue`` to enqueue the deleted resources on the FIFO and start the grace period. +* if the library runs out of resources while adding entries, it should call ``rte_rcu_qsbr_dq_reclaim`` to reclaim the resources and try the resource allocation again. + +The 'Shutdown' process needs to be shared between the application and the +client library. + +* the application should make sure that the reader threads are not using the shared data structure, unregister the reader threads from the QSBR variable before calling the client library's shutdown function. + +* client library should call ``rte_rcu_qsbr_dq_delete`` to reclaim any remaining resources and free the FIFO. + +Integrating the resource reclamation with client libraries removes the burden from +the application and makes it easy to use lock-free algorithms. + +This design has several advantages over currently known methods. + +#. Application does not need a dedicated thread to reclaim resources. Memory + reclamation happens as part of the writer thread with little impact on + performance. +#. The client library has better control over the resources. For ex: the client + library can attempt to reclaim when it has run out of resources.