From patchwork Thu Dec 6 23:25:32 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jesper Dangaard Brouer X-Patchwork-Id: 153068 Delivered-To: patch@linaro.org Received: by 2002:a2e:299d:0:0:0:0:0 with SMTP id p29-v6csp11146127ljp; Thu, 6 Dec 2018 15:26:02 -0800 (PST) X-Google-Smtp-Source: AFSGD/W0HM+OjX5HMYgJY+pu+Uftf2EhtmxzwGPqGqLbzHukdTM+wTtZTaHh1l6mC0rXQljBFBpB X-Received: by 2002:a17:902:7683:: with SMTP id m3mr11442480pll.187.1544138762427; Thu, 06 Dec 2018 15:26:02 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1544138762; cv=none; d=google.com; s=arc-20160816; b=cDrIqSNl06FDJvX2YS5R0rybvVhpWSO06+r2OPq3bd0CyVNfvWt4LcvSYh2VsKrRF/ MYLE2oE4c8MF1rTLStOLBCaZns+Ocg8lKA1GrtmGymErxFrl55Afgr6y1G3LbRJw7GQm +nV9tUFxkpOtM4c2AZ1sCK2Rt4N5tyjwRnwtJ4DiD9kq2f1IRgzeH+KMksJIpmJbKqz4 7yVOHtxCo0ev947ficAt7vNtn93CDFpEaTP/845ItQCaTx/j0vlBt5R2Mt0MVCEvse4J aX8LSYC6XT/VBTAWpkDQia8eOGHtRjq+X7eW+D2E1H/JRezQSgze3gcvg4n/7glf4Pi3 8gFA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :user-agent:references:in-reply-to:message-id:date:cc:to:from :subject; bh=mT7WV54Gg4oJwNbI63meHKsJ37ZINZ7M7JSk3WDX3fQ=; b=IWkHxOxTNeIcyjJ4LEshVt4ERUrESkM6uFBlAhFxnkE78mqIcuwzxJXXKh+RDDK2HS d+9iSh/kNqILHuJymMaMkSbO0QksuW3GHmb5Moqrs8uOrbj1lKuItJuu97D/oqOvqlJB EnzWanaZ/CCxrJlvt5zVNQoOP9obKCaMW45k8o4ZWZVmz023qKHMd+VILM2l8gFKoyWq E6G5lkVXfJ4VK/66WvidoK8fTeG6cIn58RzaM88KvBCDy+VuA+Btp0+1bZAtOLjTr5jL FPo8YJmaQnSY7RvRqMnuBcvvZDznEoqeDQWkI5Q7bCvPKRXHwcOn44LuR/4GuJGicmeA a2VQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of netdev-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=netdev-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=redhat.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id m32si1337455pld.86.2018.12.06.15.26.02; Thu, 06 Dec 2018 15:26:02 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of netdev-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of netdev-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=netdev-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726208AbeLFXZj (ORCPT + 10 others); Thu, 6 Dec 2018 18:25:39 -0500 Received: from mx1.redhat.com ([209.132.183.28]:58902 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726134AbeLFXZi (ORCPT ); Thu, 6 Dec 2018 18:25:38 -0500 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.phx2.redhat.com [10.5.11.15]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 599BC300273A; Thu, 6 Dec 2018 23:25:37 +0000 (UTC) Received: from firesoul.localdomain (ovpn-200-16.brq.redhat.com [10.40.200.16]) by smtp.corp.redhat.com (Postfix) with ESMTP id E61FC6249B; Thu, 6 Dec 2018 23:25:32 +0000 (UTC) Received: from [192.168.5.1] (localhost [IPv6:::1]) by firesoul.localdomain (Postfix) with ESMTP id 1D3CA31256FC6; Fri, 7 Dec 2018 00:25:32 +0100 (CET) Subject: [net-next PATCH RFC 1/8] page_pool: add helper functions for DMA From: Jesper Dangaard Brouer To: netdev@vger.kernel.org, "David S. Miller" , Jesper Dangaard Brouer Cc: Toke =?utf-8?q?H=C3=B8iland-J=C3=B8rgensen?= , ard.biesheuvel@linaro.org, Jason Wang , ilias.apalodimas@linaro.org, =?utf-8?b?QmrDtnJuVMO2cGVs?= , w@1wt.eu, Saeed Mahameed , mykyta.iziumtsev@gmail.com, Daniel Borkmann , Alexei Starovoitov , Tariq Toukan Date: Fri, 07 Dec 2018 00:25:32 +0100 Message-ID: <154413873204.21735.4234188123129390865.stgit@firesoul> In-Reply-To: <154413868810.21735.572808840657728172.stgit@firesoul> References: <154413868810.21735.572808840657728172.stgit@firesoul> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.15 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.46]); Thu, 06 Dec 2018 23:25:38 +0000 (UTC) Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Ilias Apalodimas Add helper functions for retreiving dma_addr_t stored in page_private and unmapping dma addresses, mapped via the page_pool API. Signed-off-by: Ilias Apalodimas Signed-off-by: Jesper Dangaard Brouer --- include/net/page_pool.h | 6 ++++++ net/core/page_pool.c | 7 +++++++ 2 files changed, 13 insertions(+) diff --git a/include/net/page_pool.h b/include/net/page_pool.h index 694d055e01ef..439f9183d4cd 100644 --- a/include/net/page_pool.h +++ b/include/net/page_pool.h @@ -111,6 +111,8 @@ struct page_pool *page_pool_create(const struct page_pool_params *params); void page_pool_destroy(struct page_pool *pool); +void page_pool_unmap_page(struct page_pool *pool, struct page *page); + /* Never call this directly, use helpers below */ void __page_pool_put_page(struct page_pool *pool, struct page *page, bool allow_direct); @@ -141,4 +143,8 @@ static inline bool is_page_pool_compiled_in(void) #endif } +static inline dma_addr_t page_pool_get_dma_addr(struct page *page) +{ + return page_private(page); +} #endif /* _NET_PAGE_POOL_H */ diff --git a/net/core/page_pool.c b/net/core/page_pool.c index 43a932cb609b..26e14a17a67c 100644 --- a/net/core/page_pool.c +++ b/net/core/page_pool.c @@ -184,6 +184,13 @@ static void __page_pool_clean_page(struct page_pool *pool, set_page_private(page, 0); } +/* unmap the page and clean our state */ +void page_pool_unmap_page(struct page_pool *pool, struct page *page) +{ + __page_pool_clean_page(pool, page); +} +EXPORT_SYMBOL(page_pool_unmap_page); + /* Return a page to the page allocator, cleaning up our state */ static void __page_pool_return_page(struct page_pool *pool, struct page *page) { From patchwork Thu Dec 6 23:25:37 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jesper Dangaard Brouer X-Patchwork-Id: 153069 Delivered-To: patch@linaro.org Received: by 2002:a2e:299d:0:0:0:0:0 with SMTP id p29-v6csp11146131ljp; Thu, 6 Dec 2018 15:26:02 -0800 (PST) X-Google-Smtp-Source: AFSGD/UUfNGKuHadUwCEZNqSMYG6AJ3797lLeqJVzjs50SeKANyjVV8yTUqYyD1M0GAgrGFM0bb6 X-Received: by 2002:a62:2fc3:: with SMTP id v186mr10629723pfv.82.1544138762715; Thu, 06 Dec 2018 15:26:02 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1544138762; cv=none; d=google.com; s=arc-20160816; b=eYgg9KWGLadPJyBavfk02fvFUuaoGhfzgGcRM+LFG4fe52p9PcLPzzOHDLUE9E0TrR 0jsivX5OX7Ap1vj+ztVr7S+nCOUJHkKZTwsrJu8QeWDPOIDuQ+0c69Psot0tme2c+kIL 4YHP1yMtG4aAd/JHX6Ss2qaed6T4iuUuZNG8QNB0hCDRFvhR1gjJMk9ICuA0LaVRzGDF O7Z/kG0UlVZIy3R1HyfYT5HEx+5YaOrBaGyO7lpdtNS43NUraOesfiISUu1vXDUnh3Tt HZoAM/65i5kwetf3MP8PV2C1Af3mTXVaM4srn87P9DTYNdLhCUaXgasF7wBN4Vs70u2r 0miw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :user-agent:references:in-reply-to:message-id:date:cc:to:from :subject; bh=AKfo1Xe2lvh0zZu3rhRvUMurzK0mtLjAzB2UvKRNEzo=; b=LA+bM8jJV1g87J/wOeHcpbADTTpIIhShO7IXuw0aH5rJI8jYbMaEIjmHKBfV0rleyt oznwIrOlbyJDJ1TmWf1LTS8DatNBF0vX7dO7gzojv1KYNPt4+oRsEWi1bIFaTrtDOOyH y5F801aAJ++tbZjd3WZcBQfsJontJh4ibdJSjc9Yoky15loM4GhSyltHCbql42FlIdr5 YZpNeqotRi0YKBQlkqTJ8YZ2CxEwcPlN7E4i9PjryMaXDbwjVIWS+58GpKviN1++Yexn j+VohEK/E3Ho6i5nUujbNlz4sF6CXc9Wm/89pA0YkwbVYn45xPDk4d4KXrJOPm2IZFvx krKg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of netdev-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=netdev-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=redhat.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id m32si1337455pld.86.2018.12.06.15.26.02; Thu, 06 Dec 2018 15:26:02 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of netdev-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of netdev-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=netdev-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726228AbeLFXZl (ORCPT + 10 others); Thu, 6 Dec 2018 18:25:41 -0500 Received: from mx1.redhat.com ([209.132.183.28]:58946 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726134AbeLFXZk (ORCPT ); Thu, 6 Dec 2018 18:25:40 -0500 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.phx2.redhat.com [10.5.11.15]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 7DF1A3001567; Thu, 6 Dec 2018 23:25:39 +0000 (UTC) Received: from firesoul.localdomain (ovpn-200-16.brq.redhat.com [10.40.200.16]) by smtp.corp.redhat.com (Postfix) with ESMTP id DA3A362671; Thu, 6 Dec 2018 23:25:38 +0000 (UTC) Received: from [192.168.5.1] (localhost [IPv6:::1]) by firesoul.localdomain (Postfix) with ESMTP id 3222F31256FC7; Fri, 7 Dec 2018 00:25:37 +0100 (CET) Subject: [net-next PATCH RFC 2/8] net: mvneta: use page pool API for sw buffer manager From: Jesper Dangaard Brouer To: netdev@vger.kernel.org, "David S. Miller" , Jesper Dangaard Brouer Cc: Toke =?utf-8?q?H=C3=B8iland-J=C3=B8rgensen?= , ard.biesheuvel@linaro.org, Jason Wang , ilias.apalodimas@linaro.org, =?utf-8?b?QmrDtnJuVMO2cGVs?= , w@1wt.eu, Saeed Mahameed , mykyta.iziumtsev@gmail.com, Daniel Borkmann , Alexei Starovoitov , Tariq Toukan Date: Fri, 07 Dec 2018 00:25:37 +0100 Message-ID: <154413873712.21735.9487067271289391052.stgit@firesoul> In-Reply-To: <154413868810.21735.572808840657728172.stgit@firesoul> References: <154413868810.21735.572808840657728172.stgit@firesoul> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.15 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.46]); Thu, 06 Dec 2018 23:25:40 +0000 (UTC) Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Ilias Apalodimas Use the page_pool api for allocations and DMA handling instead of __dev_alloc_page()/dma_map_page() and free_page()/dma_unmap_page(). The page_pool API offers buffer recycling capabilities for XDP but allocates one page per packet, unless the driver splits and manages the allocated page. Although XDP is not a part of the driver yet, the current implementation is allocating one page per packet, thus there's no performance penalty from using the API. For now pages are unmapped via page_pool_unmap_page() before packets travel into the network stack, as it doesn't have a return hook yet. Given this call cleared the page_pool state, it is safe to let the page be returned to the normal page allocator. Signed-off-by: Ilias Apalodimas Signed-off-by: Jesper Dangaard Brouer --- drivers/net/ethernet/marvell/Kconfig | 1 + drivers/net/ethernet/marvell/mvneta.c | 56 ++++++++++++++++++++++++--------- 2 files changed, 41 insertions(+), 16 deletions(-) diff --git a/drivers/net/ethernet/marvell/Kconfig b/drivers/net/ethernet/marvell/Kconfig index 3238aa7f5dac..3325abe67465 100644 --- a/drivers/net/ethernet/marvell/Kconfig +++ b/drivers/net/ethernet/marvell/Kconfig @@ -60,6 +60,7 @@ config MVNETA depends on ARCH_MVEBU || COMPILE_TEST select MVMDIO select PHYLINK + select PAGE_POOL ---help--- This driver supports the network interface units in the Marvell ARMADA XP, ARMADA 370, ARMADA 38x and diff --git a/drivers/net/ethernet/marvell/mvneta.c b/drivers/net/ethernet/marvell/mvneta.c index 5bfd349bf41a..2354421fe96f 100644 --- a/drivers/net/ethernet/marvell/mvneta.c +++ b/drivers/net/ethernet/marvell/mvneta.c @@ -33,6 +33,7 @@ #include #include #include "mvneta_bm.h" +#include #include #include #include @@ -624,6 +625,9 @@ struct mvneta_rx_queue { struct sk_buff *skb; int left_size; + /* page pool */ + struct page_pool *page_pool; + /* error counters */ u32 skb_alloc_err; u32 refill_err; @@ -1813,17 +1817,11 @@ static int mvneta_rx_refill(struct mvneta_port *pp, dma_addr_t phys_addr; struct page *page; - page = __dev_alloc_page(gfp_mask); + page = page_pool_dev_alloc_pages(rxq->page_pool); if (!page) return -ENOMEM; - /* map page for use */ - phys_addr = dma_map_page(pp->dev->dev.parent, page, 0, PAGE_SIZE, - DMA_FROM_DEVICE); - if (unlikely(dma_mapping_error(pp->dev->dev.parent, phys_addr))) { - __free_page(page); - return -ENOMEM; - } + phys_addr = page_pool_get_dma_addr(page); phys_addr += pp->rx_offset_correction; mvneta_rx_desc_fill(rx_desc, phys_addr, page, rxq); @@ -1892,10 +1890,11 @@ static void mvneta_rxq_drop_pkts(struct mvneta_port *pp, if (!data || !(rx_desc->buf_phys_addr)) continue; - dma_unmap_page(pp->dev->dev.parent, rx_desc->buf_phys_addr, - PAGE_SIZE, DMA_FROM_DEVICE); - __free_page(data); + page_pool_put_page(rxq->page_pool, data, false); } + + if (rxq->page_pool) + page_pool_destroy(rxq->page_pool); } static inline @@ -2010,8 +2009,7 @@ static int mvneta_rx_swbm(struct napi_struct *napi, skb_add_rx_frag(rxq->skb, frag_num, page, frag_offset, frag_size, PAGE_SIZE); - dma_unmap_page(dev->dev.parent, phys_addr, - PAGE_SIZE, DMA_FROM_DEVICE); + page_pool_unmap_page(rxq->page_pool, page); rxq->left_size -= frag_size; } } else { @@ -2041,8 +2039,7 @@ static int mvneta_rx_swbm(struct napi_struct *napi, frag_offset, frag_size, PAGE_SIZE); - dma_unmap_page(dev->dev.parent, phys_addr, - PAGE_SIZE, DMA_FROM_DEVICE); + page_pool_unmap_page(rxq->page_pool, page); rxq->left_size -= frag_size; } @@ -2828,11 +2825,37 @@ static int mvneta_poll(struct napi_struct *napi, int budget) return rx_done; } +static int mvneta_create_page_pool(struct mvneta_port *pp, + struct mvneta_rx_queue *rxq, int num) +{ + struct page_pool_params pp_params = { 0 }; + int err = 0; + + pp_params.order = 0; + /* internal DMA mapping in page_pool */ + pp_params.flags = PP_FLAG_DMA_MAP; + pp_params.pool_size = num; + pp_params.nid = NUMA_NO_NODE; + pp_params.dev = pp->dev->dev.parent; + pp_params.dma_dir = DMA_FROM_DEVICE; + + rxq->page_pool = page_pool_create(&pp_params); + if (IS_ERR(rxq->page_pool)) { + err = PTR_ERR(rxq->page_pool); + rxq->page_pool = NULL; + } + + return err; +} + /* Handle rxq fill: allocates rxq skbs; called when initializing a port */ static int mvneta_rxq_fill(struct mvneta_port *pp, struct mvneta_rx_queue *rxq, int num) { - int i; + int i = 0; + + if (mvneta_create_page_pool(pp, rxq, num)) + goto out; for (i = 0; i < num; i++) { memset(rxq->descs + i, 0, sizeof(struct mvneta_rx_desc)); @@ -2848,6 +2871,7 @@ static int mvneta_rxq_fill(struct mvneta_port *pp, struct mvneta_rx_queue *rxq, /* Add this number of RX descriptors as non occupied (ready to * get packets) */ +out: mvneta_rxq_non_occup_desc_add(pp, rxq, i); return i; From patchwork Thu Dec 6 23:25:47 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jesper Dangaard Brouer X-Patchwork-Id: 153070 Delivered-To: patch@linaro.org Received: by 2002:a2e:299d:0:0:0:0:0 with SMTP id p29-v6csp11146159ljp; Thu, 6 Dec 2018 15:26:04 -0800 (PST) X-Google-Smtp-Source: AFSGD/Vw8crJXD2qSHmKcXVAUwZ+rdaT+83x7WKOrUOzxXwrjjOp8rb5LwyVTUgGERT8Xg0kroUz X-Received: by 2002:a63:9c1a:: with SMTP id f26mr7324522pge.381.1544138764149; Thu, 06 Dec 2018 15:26:04 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1544138764; cv=none; d=google.com; s=arc-20160816; b=QVHdNFVmSHW19+7hGIbgEHhk1lIu/pt5jDL9u/mkKRyR6otPuw9TMpZBuHeCbP8BKG AcyNb4pmiEdD3gBFGL62yw71lcwPlGaUttIfjK5mkP/KMhD7Nd6A1EputCiUxX94rOml 16/FxcJVxh3is6jUc8DoL8M5tItDjzGX0QijResE7pvFLsA741Llyzs/HyrtN6Qzms9W IwzXRKRpAzd+/QIFfJq2h4qW7U5lRfD3Qsi83/bf1+u1aaR3DKgNU5+WfJVUpTLwAwHw 59e/f3rQUtDUHw3UCuYMKQBRjg1W5505OONGWpgzFqDqhwATFsrgpNeZNBFYb2Yd2jBP U6iQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :user-agent:references:in-reply-to:message-id:date:cc:to:from :subject; bh=ZzUvN/YSbdJLYjLzDsUOz7nslSDIuuMDEEq5nf9wMLE=; b=YEak36piAVTl+/0LsfQXGF0jv5n0fE4AmlAzQLMYV+jyuBLNnb+6ZReozDIYB4n1iU W22KdgaokAWrpuILEmx3s3fePhQIq7HMcSapDXVL75tire7oZUJeOzc97K/QUNEkIVVA Sn04Dk/po3uCNvSi8A5t1zUe3EfGdrjcpR49reH+eLIyqCAsyHtwgQMyEK+cA3+uXfUU fdvnYUupzkr4XZSgggU6xmnSezaq7vurO7bpatLn2Qcdy8htg7fXtJmKgbNLXwuYT2N7 1J8MIdSq4Ndr2SQLsaQjxiVH+9l/9YoBNRKMCH4t2O4GUgzR4k2BoJcnB+V4ZzftTYaU ullw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of netdev-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=netdev-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=redhat.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id m32si1337455pld.86.2018.12.06.15.26.03; Thu, 06 Dec 2018 15:26:04 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of netdev-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of netdev-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=netdev-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726253AbeLFXZz (ORCPT + 10 others); Thu, 6 Dec 2018 18:25:55 -0500 Received: from mx1.redhat.com ([209.132.183.28]:9085 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726069AbeLFXZz (ORCPT ); Thu, 6 Dec 2018 18:25:55 -0500 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id AF6AE30821A4; Thu, 6 Dec 2018 23:25:54 +0000 (UTC) Received: from firesoul.localdomain (ovpn-200-16.brq.redhat.com [10.40.200.16]) by smtp.corp.redhat.com (Postfix) with ESMTP id 303755D9D6; Thu, 6 Dec 2018 23:25:48 +0000 (UTC) Received: from [192.168.5.1] (localhost [IPv6:::1]) by firesoul.localdomain (Postfix) with ESMTP id 5BE6830631A99; Fri, 7 Dec 2018 00:25:47 +0100 (CET) Subject: [net-next PATCH RFC 4/8] net: core: add recycle capabilities on skbs via page_pool API From: Jesper Dangaard Brouer To: netdev@vger.kernel.org, "David S. Miller" , Jesper Dangaard Brouer Cc: Toke =?utf-8?q?H=C3=B8iland-J=C3=B8rgensen?= , ard.biesheuvel@linaro.org, Jason Wang , ilias.apalodimas@linaro.org, =?utf-8?b?QmrDtnJuVMO2cGVs?= , w@1wt.eu, Saeed Mahameed , mykyta.iziumtsev@gmail.com, Daniel Borkmann , Alexei Starovoitov , Tariq Toukan Date: Fri, 07 Dec 2018 00:25:47 +0100 Message-ID: <154413874729.21735.10644578158550468689.stgit@firesoul> In-Reply-To: <154413868810.21735.572808840657728172.stgit@firesoul> References: <154413868810.21735.572808840657728172.stgit@firesoul> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.47]); Thu, 06 Dec 2018 23:25:54 +0000 (UTC) Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Ilias Apalodimas This patch is changing struct sk_buff, and is thus per-definition controversial. Place a new member 'mem_info' of type struct xdp_mem_info, just after members (flags) head_frag and pfmemalloc, And not in between headers_start/end to ensure skb_copy() and pskb_copy() work as-is. Copying mem_info during skb_clone() is required. This makes sure that pages are correctly freed or recycled during the altered skb_free_head() invocation. The 'mem_info' name is chosen as this is not strictly tied to XDP, although the XDP return infrastructure is used. As a future plan, we could introduce a __u8 flags member to xdp_mem_info and move flags head_frag and pfmemalloc into this area. Signed-off-by: Ilias Apalodimas Signed-off-by: Jesper Dangaard Brouer --- include/linux/skbuff.h | 6 +++++- include/net/xdp.h | 1 + net/core/skbuff.c | 7 +++++++ net/core/xdp.c | 6 ++++++ 4 files changed, 19 insertions(+), 1 deletion(-) diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h index 7dcfb5591dc3..95dac0ba6947 100644 --- a/include/linux/skbuff.h +++ b/include/linux/skbuff.h @@ -40,6 +40,7 @@ #include #include #include +#include /* The interface for checksum offload between the stack and networking drivers * is as follows... @@ -744,6 +745,10 @@ struct sk_buff { head_frag:1, xmit_more:1, pfmemalloc:1; + /* TODO: Future idea, extend mem_info with __u8 flags, and + * move bits head_frag and pfmemalloc there. + */ + struct xdp_mem_info mem_info; /* fields enclosed in headers_start/headers_end are copied * using a single memcpy() in __copy_skb_header() @@ -827,7 +832,6 @@ struct sk_buff { #ifdef CONFIG_NETWORK_SECMARK __u32 secmark; #endif - union { __u32 mark; __u32 reserved_tailroom; diff --git a/include/net/xdp.h b/include/net/xdp.h index 5c33b9e0efab..4a0ca7a3d5e5 100644 --- a/include/net/xdp.h +++ b/include/net/xdp.h @@ -128,6 +128,7 @@ struct xdp_frame *convert_to_xdp_frame(struct xdp_buff *xdp) void xdp_return_frame(struct xdp_frame *xdpf); void xdp_return_frame_rx_napi(struct xdp_frame *xdpf); void xdp_return_buff(struct xdp_buff *xdp); +void xdp_return_skb_page(void *data, struct xdp_mem_info *mem_info); int xdp_rxq_info_reg(struct xdp_rxq_info *xdp_rxq, struct net_device *dev, u32 queue_index); diff --git a/net/core/skbuff.c b/net/core/skbuff.c index b4ee5c8b928f..71aca186e44c 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -70,6 +70,7 @@ #include #include #include +#include #include #include @@ -544,6 +545,11 @@ static void skb_free_head(struct sk_buff *skb) { unsigned char *head = skb->head; + if (skb->mem_info.type == MEM_TYPE_PAGE_POOL) { + xdp_return_skb_page(head, &skb->mem_info); + return; + } + if (skb->head_frag) skb_free_frag(head); else @@ -859,6 +865,7 @@ static struct sk_buff *__skb_clone(struct sk_buff *n, struct sk_buff *skb) n->nohdr = 0; n->peeked = 0; C(pfmemalloc); + C(mem_info); n->destructor = NULL; C(tail); C(end); diff --git a/net/core/xdp.c b/net/core/xdp.c index e79526314864..1703be4c2611 100644 --- a/net/core/xdp.c +++ b/net/core/xdp.c @@ -379,6 +379,12 @@ void xdp_return_buff(struct xdp_buff *xdp) } EXPORT_SYMBOL_GPL(xdp_return_buff); +void xdp_return_skb_page(void *data, struct xdp_mem_info *mem_info) +{ + __xdp_return(data, mem_info, false, 0); +} +EXPORT_SYMBOL(xdp_return_skb_page); + int xdp_attachment_query(struct xdp_attachment_info *info, struct netdev_bpf *bpf) { From patchwork Thu Dec 6 23:25:52 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jesper Dangaard Brouer X-Patchwork-Id: 153071 Delivered-To: patch@linaro.org Received: by 2002:a2e:299d:0:0:0:0:0 with SMTP id p29-v6csp11146166ljp; Thu, 6 Dec 2018 15:26:04 -0800 (PST) X-Google-Smtp-Source: AFSGD/XygOqpOmWu2Mh1LDUN1dEGVbwJxhJBa+bMNWacQe6B3m0D9ghYpuyLivzeebZ8ZCy+CfCR X-Received: by 2002:a17:902:7443:: with SMTP id e3mr30158642plt.304.1544138764443; Thu, 06 Dec 2018 15:26:04 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1544138764; cv=none; d=google.com; s=arc-20160816; b=s/MzFZ/FiCWg3M0O0EWl1mQ8oB+Lmo7+AZFo+iK59P9DN/3VRQAa6SCaSsjfFvGm4R hCc+ZT1qKmbjgPTe8hmX8EKBl2o5JDGWX9zmqOUl/b3OzWamzVypKiG4SnRRkX7gi5Rb dbQ/W6hPSPB5Bsw//4lzlPa0iHXw0cXuj+oV0rA/aK2aD7m4yjMyLnwvmU2QS2byAZYb 3OCtlMQseh7zLHIjNLkCXZCA4Uo/ulujA+UIg1+D0EBV7HiBYzl6Fsb95J3IhrDrHj3h leXLGxrKx899tS9PTE+X9YyhoMFU/i6P9Hr0OND+Po00OOK4MHeMVRCZqn/8lfClz99k PdXw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :user-agent:references:in-reply-to:message-id:date:cc:to:from :subject; bh=0ah9YEmt2ZfDRlAVK9QbneEVgUGgc9d1Ug0P/hAumAE=; b=zX7uRiCb2Z2L58+RxNKqxYV8VdSqUjziKlizMLr4KjB9VTIOWbW5KrF0D4bW1yQwE2 PfIHIn3JdOd7LUPyFOWk/eEWaNBGVPbY6YFn2zrOuDKv1lhpErdIopJxFCAWN1KBDPkB OUyPRrd4LtSJhBDdpTgQz2tjoz6jR7U3PoHfUGhQbaHPV8v6uIca1UrtWqcxMB9iJpyM W15TXoQ0kJuc3DSSn5xVMLFx0PhcI/1ENuDjvkp3pRmRGvFK+i1CvAYkolhlXY5bIORF B4hxgzgPc8okBxtpCxuozmEGJgHsGZO9wzyHG96ZvYTzX13dP3UlsebiKLWZ6wQBmbjA F/Iw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of netdev-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=netdev-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=redhat.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id m32si1337455pld.86.2018.12.06.15.26.04; Thu, 06 Dec 2018 15:26:04 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of netdev-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of netdev-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=netdev-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726260AbeLFXZ6 (ORCPT + 10 others); Thu, 6 Dec 2018 18:25:58 -0500 Received: from mx1.redhat.com ([209.132.183.28]:59080 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726069AbeLFXZ5 (ORCPT ); Thu, 6 Dec 2018 18:25:57 -0500 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 8FCFD3003B34; Thu, 6 Dec 2018 23:25:56 +0000 (UTC) Received: from firesoul.localdomain (ovpn-200-16.brq.redhat.com [10.40.200.16]) by smtp.corp.redhat.com (Postfix) with ESMTP id 427E465F62; Thu, 6 Dec 2018 23:25:53 +0000 (UTC) Received: from [192.168.5.1] (localhost [IPv6:::1]) by firesoul.localdomain (Postfix) with ESMTP id 6E3CD31256FC6; Fri, 7 Dec 2018 00:25:52 +0100 (CET) Subject: [net-next PATCH RFC 5/8] net: mvneta: remove copybreak, prefetch and use build_skb From: Jesper Dangaard Brouer To: netdev@vger.kernel.org, "David S. Miller" , Jesper Dangaard Brouer Cc: Toke =?utf-8?q?H=C3=B8iland-J=C3=B8rgensen?= , ard.biesheuvel@linaro.org, Jason Wang , ilias.apalodimas@linaro.org, =?utf-8?b?QmrDtnJuVMO2cGVs?= , w@1wt.eu, Saeed Mahameed , mykyta.iziumtsev@gmail.com, Daniel Borkmann , Alexei Starovoitov , Tariq Toukan Date: Fri, 07 Dec 2018 00:25:52 +0100 Message-ID: <154413875238.21735.7746697931250893385.stgit@firesoul> In-Reply-To: <154413868810.21735.572808840657728172.stgit@firesoul> References: <154413868810.21735.572808840657728172.stgit@firesoul> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.46]); Thu, 06 Dec 2018 23:25:56 +0000 (UTC) Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Ilias Apalodimas The driver memcpy for packets < 256b and it's recycle tricks are not needed anymore. As previous patch introduces buffer recycling using the page_pool API (although recycling doesn't get fully activated in this patch). After this switch to using build_skb(). This patch implicit fixes a driver bug where the memory is copied prior to it's syncing for the CPU, in the < 256b case (as this case is removed). We also remove the data prefetch completely. The original driver had the prefetch misplaced before any dma_sync operations took place. Based on Jesper's analysis even if the prefetch is placed after any DMA sync ops it ends up hurting performance. Signed-off-by: Ilias Apalodimas Signed-off-by: Jesper Dangaard Brouer --- drivers/net/ethernet/marvell/mvneta.c | 81 +++++++++------------------------ 1 file changed, 22 insertions(+), 59 deletions(-) diff --git a/drivers/net/ethernet/marvell/mvneta.c b/drivers/net/ethernet/marvell/mvneta.c index 2354421fe96f..78f1fcdc1f00 100644 --- a/drivers/net/ethernet/marvell/mvneta.c +++ b/drivers/net/ethernet/marvell/mvneta.c @@ -643,7 +643,6 @@ static int txq_number = 8; static int rxq_def; static int rx_copybreak __read_mostly = 256; -static int rx_header_size __read_mostly = 128; /* HW BM need that each port be identify by a unique ID */ static int global_port_id; @@ -1823,7 +1822,7 @@ static int mvneta_rx_refill(struct mvneta_port *pp, phys_addr = page_pool_get_dma_addr(page); - phys_addr += pp->rx_offset_correction; + phys_addr += pp->rx_offset_correction + NET_SKB_PAD; mvneta_rx_desc_fill(rx_desc, phys_addr, page, rxq); return 0; } @@ -1944,14 +1943,12 @@ static int mvneta_rx_swbm(struct napi_struct *napi, struct page *page; dma_addr_t phys_addr; u32 rx_status, index; - int rx_bytes, skb_size, copy_size; - int frag_num, frag_size, frag_offset; + int frag_num, frag_size; + int rx_bytes; index = rx_desc - rxq->descs; page = (struct page *)rxq->buf_virt_addr[index]; data = page_address(page); - /* Prefetch header */ - prefetch(data); phys_addr = rx_desc->buf_phys_addr; rx_status = rx_desc->status; @@ -1969,49 +1966,25 @@ static int mvneta_rx_swbm(struct napi_struct *napi, rx_bytes = rx_desc->data_size - (ETH_FCS_LEN + MVNETA_MH_SIZE); - /* Allocate small skb for each new packet */ - skb_size = max(rx_copybreak, rx_header_size); - rxq->skb = netdev_alloc_skb_ip_align(dev, skb_size); - if (unlikely(!rxq->skb)) { - netdev_err(dev, - "Can't allocate skb on queue %d\n", - rxq->id); - dev->stats.rx_dropped++; - rxq->skb_alloc_err++; - continue; - } - copy_size = min(skb_size, rx_bytes); - - /* Copy data from buffer to SKB, skip Marvell header */ - memcpy(rxq->skb->data, data + MVNETA_MH_SIZE, - copy_size); - skb_put(rxq->skb, copy_size); - rxq->left_size = rx_bytes - copy_size; - mvneta_rx_csum(pp, rx_status, rxq->skb); - if (rxq->left_size == 0) { - int size = copy_size + MVNETA_MH_SIZE; - - dma_sync_single_range_for_cpu(dev->dev.parent, - phys_addr, 0, - size, - DMA_FROM_DEVICE); + dma_sync_single_range_for_cpu(dev->dev.parent, + phys_addr, 0, + rx_bytes, + DMA_FROM_DEVICE); - /* leave the descriptor and buffer untouched */ - } else { - /* refill descriptor with new buffer later */ - rx_desc->buf_phys_addr = 0; + rxq->skb = build_skb(data, PAGE_SIZE); + if (!rxq->skb) + break; - frag_num = 0; - frag_offset = copy_size + MVNETA_MH_SIZE; - frag_size = min(rxq->left_size, - (int)(PAGE_SIZE - frag_offset)); - skb_add_rx_frag(rxq->skb, frag_num, page, - frag_offset, frag_size, - PAGE_SIZE); - page_pool_unmap_page(rxq->page_pool, page); - rxq->left_size -= frag_size; - } + rx_desc->buf_phys_addr = 0; + frag_num = 0; + skb_reserve(rxq->skb, MVNETA_MH_SIZE + NET_SKB_PAD); + skb_put(rxq->skb, rx_bytes < PAGE_SIZE ? rx_bytes : + PAGE_SIZE); + mvneta_rx_csum(pp, rx_status, rxq->skb); + page_pool_unmap_page(rxq->page_pool, page); + rxq->left_size = rx_bytes < PAGE_SIZE ? 0 : rx_bytes - + PAGE_SIZE; } else { /* Middle or Last descriptor */ if (unlikely(!rxq->skb)) { @@ -2019,24 +1992,14 @@ static int mvneta_rx_swbm(struct napi_struct *napi, rx_status); continue; } - if (!rxq->left_size) { - /* last descriptor has only FCS */ - /* and can be discarded */ - dma_sync_single_range_for_cpu(dev->dev.parent, - phys_addr, 0, - ETH_FCS_LEN, - DMA_FROM_DEVICE); - /* leave the descriptor and buffer untouched */ - } else { + if (rxq->left_size) { /* refill descriptor with new buffer later */ rx_desc->buf_phys_addr = 0; frag_num = skb_shinfo(rxq->skb)->nr_frags; - frag_offset = 0; - frag_size = min(rxq->left_size, - (int)(PAGE_SIZE - frag_offset)); + frag_size = min(rxq->left_size, (int)PAGE_SIZE); skb_add_rx_frag(rxq->skb, frag_num, page, - frag_offset, frag_size, + 0, frag_size, PAGE_SIZE); page_pool_unmap_page(rxq->page_pool, page);