From patchwork Wed Jul 28 09:38:07 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lorenzo Bianconi X-Patchwork-Id: 488476 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-19.7 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, INCLUDES_CR_TRAILER, INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E7C74C432BE for ; Wed, 28 Jul 2021 09:38:55 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id CA7EE60FD8 for ; Wed, 28 Jul 2021 09:38:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235629AbhG1Ji4 (ORCPT ); Wed, 28 Jul 2021 05:38:56 -0400 Received: from mail.kernel.org ([198.145.29.99]:51446 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235606AbhG1Jiz (ORCPT ); Wed, 28 Jul 2021 05:38:55 -0400 Received: by mail.kernel.org (Postfix) with ESMTPSA id 36EEC60FC0; Wed, 28 Jul 2021 09:38:51 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1627465134; bh=7rec31JTsnV/0vJfrcV5ZsizOBgtcA3Ty6UT24MBJtg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=UmZoaKK96aP1+F1FjeKw5TpCU75VYjmvsWpqqAdpBZTG8Z7ICCDbBSlRcAawOKHqJ gE0WHr/8i/F3ocbCpk89Z6KmM2bQ1PXmUEaqaVm/m0JgPDHmNRCWxtHABlCMRx2LY4 m/cGSo9ra0QOWI0e6Fl8Zy4gr3HzG8gv1Ukcm9nxMBTWCw9xCOUs+A5P9rTluIoXtU s1tsTp+lQaQTTUEioc8lgROomZ5qCo2OCkA3KM31n1hEUzqv70UZkpNIqPelfGGE8r D32vXSNQJBjSqg3E0+T9fdmpnIg13lXijUHyOPXPzLdb/zcnG56r0RB38fOkh7+GTC kxYHl/5DhMd6A== From: Lorenzo Bianconi To: bpf@vger.kernel.org, netdev@vger.kernel.org Cc: lorenzo.bianconi@redhat.com, davem@davemloft.net, kuba@kernel.org, ast@kernel.org, daniel@iogearbox.net, shayagr@amazon.com, john.fastabend@gmail.com, dsahern@kernel.org, brouer@redhat.com, echaudro@redhat.com, jasowang@redhat.com, alexander.duyck@gmail.com, saeed@kernel.org, maciej.fijalkowski@intel.com, magnus.karlsson@intel.com, tirthendu.sarkar@intel.com, toke@redhat.com Subject: [PATCH v10 bpf-next 02/18] xdp: introduce flags field in xdp_buff/xdp_frame Date: Wed, 28 Jul 2021 11:38:07 +0200 Message-Id: X-Mailer: git-send-email 2.31.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org Introduce flags field in xdp_frame and xdp_buffer data structures to define additional buffer features. At the moment the only supported buffer feature is multi-buffer bit (mb). Multi-buffer bit is used to specify if this is a linear buffer (mb = 0) or a multi-buffer frame (mb = 1). In the latter case the driver is expected to initialize the skb_shared_info structure at the end of the first buffer to link together subsequent buffers belonging to the same frame. Signed-off-by: Lorenzo Bianconi --- include/net/xdp.h | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) diff --git a/include/net/xdp.h b/include/net/xdp.h index ad5b02dcb6f4..ed5ea784fd45 100644 --- a/include/net/xdp.h +++ b/include/net/xdp.h @@ -66,6 +66,10 @@ struct xdp_txq_info { struct net_device *dev; }; +enum xdp_buff_flags { + XDP_FLAGS_MULTI_BUFF = BIT(0), /* non-linear xdp buff */ +}; + struct xdp_buff { void *data; void *data_end; @@ -74,13 +78,30 @@ struct xdp_buff { struct xdp_rxq_info *rxq; struct xdp_txq_info *txq; u32 frame_sz; /* frame size to deduce data_hard_end/reserved tailroom*/ + u16 flags; /* supported values defined in xdp_flags */ }; +static __always_inline bool xdp_buff_is_mb(struct xdp_buff *xdp) +{ + return !!(xdp->flags & XDP_FLAGS_MULTI_BUFF); +} + +static __always_inline void xdp_buff_set_mb(struct xdp_buff *xdp) +{ + xdp->flags |= XDP_FLAGS_MULTI_BUFF; +} + +static __always_inline void xdp_buff_clear_mb(struct xdp_buff *xdp) +{ + xdp->flags &= ~XDP_FLAGS_MULTI_BUFF; +} + static __always_inline void xdp_init_buff(struct xdp_buff *xdp, u32 frame_sz, struct xdp_rxq_info *rxq) { xdp->frame_sz = frame_sz; xdp->rxq = rxq; + xdp->flags = 0; } static __always_inline void @@ -122,8 +143,14 @@ struct xdp_frame { */ struct xdp_mem_info mem; struct net_device *dev_rx; /* used by cpumap */ + u16 flags; /* supported values defined in xdp_flags */ }; +static __always_inline bool xdp_frame_is_mb(struct xdp_frame *frame) +{ + return !!(frame->flags & XDP_FLAGS_MULTI_BUFF); +} + #define XDP_BULK_QUEUE_SIZE 16 struct xdp_frame_bulk { int count; @@ -180,6 +207,7 @@ void xdp_convert_frame_to_buff(struct xdp_frame *frame, struct xdp_buff *xdp) xdp->data_end = frame->data + frame->len; xdp->data_meta = frame->data - frame->metasize; xdp->frame_sz = frame->frame_sz; + xdp->flags = frame->flags; } static inline @@ -206,6 +234,7 @@ int xdp_update_frame_from_buff(struct xdp_buff *xdp, xdp_frame->headroom = headroom - sizeof(*xdp_frame); xdp_frame->metasize = metasize; xdp_frame->frame_sz = xdp->frame_sz; + xdp_frame->flags = xdp->flags; return 0; } From patchwork Wed Jul 28 09:38:09 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lorenzo Bianconi X-Patchwork-Id: 488475 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-19.7 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, INCLUDES_CR_TRAILER, INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C237FC432BE for ; Wed, 28 Jul 2021 09:39:04 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id A7FC860FC0 for ; Wed, 28 Jul 2021 09:39:04 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235656AbhG1JjF (ORCPT ); Wed, 28 Jul 2021 05:39:05 -0400 Received: from mail.kernel.org ([198.145.29.99]:51524 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234989AbhG1JjE (ORCPT ); Wed, 28 Jul 2021 05:39:04 -0400 Received: by mail.kernel.org (Postfix) with ESMTPSA id B138060FD8; Wed, 28 Jul 2021 09:38:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1627465142; bh=iCaQEnG389bALUGd41KNWDp1ItcsGvIsXjEJYSRrZRo=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=DaYwRv16h97tpuNOO//A5oLVHYlcrXLAwfkoB8F5W5yBT7UyRqPXPbV+19gxECKjW BZTNTmjIFhw45VwW1bnVpxJwPSJogDB1xWrLNeIBeRCF1DvT4CrmjoBm3SuTFlhGbC J/2feVyL7/EZO+oSC1SRp2oO1/jXSvZrCi8sKPRgzq2tnPaAC3VsBuUb/EqI89Q8U5 MIa2GgD4cwVaOpIq0U1nbLu1K5CaCHxlBVWY9Ti9TeMSIZnqGbvhMeaJy0j8zNz+It qVQpS1x38MeTSmYlz2c2GzEx9wVzsYuDtuAYhx4CPcENyFKJcbWYmfmXHSFidYm++T gcpyR6XHSQgwQ== From: Lorenzo Bianconi To: bpf@vger.kernel.org, netdev@vger.kernel.org Cc: lorenzo.bianconi@redhat.com, davem@davemloft.net, kuba@kernel.org, ast@kernel.org, daniel@iogearbox.net, shayagr@amazon.com, john.fastabend@gmail.com, dsahern@kernel.org, brouer@redhat.com, echaudro@redhat.com, jasowang@redhat.com, alexander.duyck@gmail.com, saeed@kernel.org, maciej.fijalkowski@intel.com, magnus.karlsson@intel.com, tirthendu.sarkar@intel.com, toke@redhat.com Subject: [PATCH v10 bpf-next 04/18] net: mvneta: simplify mvneta_swbm_add_rx_fragment management Date: Wed, 28 Jul 2021 11:38:09 +0200 Message-Id: X-Mailer: git-send-email 2.31.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org Relying on xdp mb bit, remove skb_shared_info structure allocated on the stack in mvneta_rx_swbm routine and simplify mvneta_swbm_add_rx_fragment accessing skb_shared_info in the xdp_buff structure directly. There is no performance penalty in this approach since mvneta_swbm_add_rx_fragment is run just for multi-buff use-case. Signed-off-by: Lorenzo Bianconi --- drivers/net/ethernet/marvell/mvneta.c | 42 ++++++++++----------------- 1 file changed, 15 insertions(+), 27 deletions(-) diff --git a/drivers/net/ethernet/marvell/mvneta.c b/drivers/net/ethernet/marvell/mvneta.c index 14a4ec3c36b0..60fc2d3ca619 100644 --- a/drivers/net/ethernet/marvell/mvneta.c +++ b/drivers/net/ethernet/marvell/mvneta.c @@ -2032,9 +2032,9 @@ int mvneta_rx_refill_queue(struct mvneta_port *pp, struct mvneta_rx_queue *rxq) static void mvneta_xdp_put_buff(struct mvneta_port *pp, struct mvneta_rx_queue *rxq, - struct xdp_buff *xdp, struct skb_shared_info *sinfo, - int sync_len) + struct xdp_buff *xdp, int sync_len) { + struct skb_shared_info *sinfo = xdp_get_shared_info_from_buff(xdp); int i; if (likely(!xdp_buff_is_mb(xdp))) @@ -2182,7 +2182,6 @@ mvneta_run_xdp(struct mvneta_port *pp, struct mvneta_rx_queue *rxq, struct bpf_prog *prog, struct xdp_buff *xdp, u32 frame_sz, struct mvneta_stats *stats) { - struct skb_shared_info *sinfo = xdp_get_shared_info_from_buff(xdp); unsigned int len, data_len, sync; u32 ret, act; @@ -2203,7 +2202,7 @@ mvneta_run_xdp(struct mvneta_port *pp, struct mvneta_rx_queue *rxq, err = xdp_do_redirect(pp->dev, xdp, prog); if (unlikely(err)) { - mvneta_xdp_put_buff(pp, rxq, xdp, sinfo, sync); + mvneta_xdp_put_buff(pp, rxq, xdp, sync); ret = MVNETA_XDP_DROPPED; } else { ret = MVNETA_XDP_REDIR; @@ -2214,7 +2213,7 @@ mvneta_run_xdp(struct mvneta_port *pp, struct mvneta_rx_queue *rxq, case XDP_TX: ret = mvneta_xdp_xmit_back(pp, xdp); if (ret != MVNETA_XDP_TX) - mvneta_xdp_put_buff(pp, rxq, xdp, sinfo, sync); + mvneta_xdp_put_buff(pp, rxq, xdp, sync); break; default: bpf_warn_invalid_xdp_action(act); @@ -2223,7 +2222,7 @@ mvneta_run_xdp(struct mvneta_port *pp, struct mvneta_rx_queue *rxq, trace_xdp_exception(pp->dev, prog, act); fallthrough; case XDP_DROP: - mvneta_xdp_put_buff(pp, rxq, xdp, sinfo, sync); + mvneta_xdp_put_buff(pp, rxq, xdp, sync); ret = MVNETA_XDP_DROPPED; stats->xdp_drop++; break; @@ -2275,9 +2274,9 @@ mvneta_swbm_add_rx_fragment(struct mvneta_port *pp, struct mvneta_rx_desc *rx_desc, struct mvneta_rx_queue *rxq, struct xdp_buff *xdp, int *size, - struct skb_shared_info *xdp_sinfo, struct page *page) { + struct skb_shared_info *sinfo = xdp_get_shared_info_from_buff(xdp); struct net_device *dev = pp->dev; enum dma_data_direction dma_dir; int data_len, len; @@ -2295,8 +2294,11 @@ mvneta_swbm_add_rx_fragment(struct mvneta_port *pp, len, dma_dir); rx_desc->buf_phys_addr = 0; - if (data_len > 0 && xdp_sinfo->nr_frags < MAX_SKB_FRAGS) { - skb_frag_t *frag = &xdp_sinfo->frags[xdp_sinfo->nr_frags++]; + if (!xdp_buff_is_mb(xdp)) + sinfo->nr_frags = 0; + + if (data_len > 0 && sinfo->nr_frags < MAX_SKB_FRAGS) { + skb_frag_t *frag = &sinfo->frags[sinfo->nr_frags++]; skb_frag_off_set(frag, pp->rx_offset_correction); skb_frag_size_set(frag, data_len); @@ -2307,16 +2309,6 @@ mvneta_swbm_add_rx_fragment(struct mvneta_port *pp, } else { page_pool_put_full_page(rxq->page_pool, page, true); } - - /* last fragment */ - if (len == *size) { - struct skb_shared_info *sinfo; - - sinfo = xdp_get_shared_info_from_buff(xdp); - sinfo->nr_frags = xdp_sinfo->nr_frags; - memcpy(sinfo->frags, xdp_sinfo->frags, - sinfo->nr_frags * sizeof(skb_frag_t)); - } *size -= len; } @@ -2368,7 +2360,6 @@ static int mvneta_rx_swbm(struct napi_struct *napi, { int rx_proc = 0, rx_todo, refill, size = 0; struct net_device *dev = pp->dev; - struct skb_shared_info sinfo; struct mvneta_stats ps = {}; struct bpf_prog *xdp_prog; u32 desc_status, frame_sz; @@ -2377,8 +2368,6 @@ static int mvneta_rx_swbm(struct napi_struct *napi, xdp_init_buff(&xdp_buf, PAGE_SIZE, &rxq->xdp_rxq); xdp_buf.data_hard_start = NULL; - sinfo.nr_frags = 0; - /* Get number of received packets */ rx_todo = mvneta_rxq_busy_desc_num_get(pp, rxq); @@ -2420,7 +2409,7 @@ static int mvneta_rx_swbm(struct napi_struct *napi, } mvneta_swbm_add_rx_fragment(pp, rx_desc, rxq, &xdp_buf, - &size, &sinfo, page); + &size, page); } /* Middle or Last descriptor */ if (!(rx_status & MVNETA_RXD_LAST_DESC)) @@ -2428,7 +2417,7 @@ static int mvneta_rx_swbm(struct napi_struct *napi, continue; if (size) { - mvneta_xdp_put_buff(pp, rxq, &xdp_buf, &sinfo, -1); + mvneta_xdp_put_buff(pp, rxq, &xdp_buf, -1); goto next; } @@ -2440,7 +2429,7 @@ static int mvneta_rx_swbm(struct napi_struct *napi, if (IS_ERR(skb)) { struct mvneta_pcpu_stats *stats = this_cpu_ptr(pp->stats); - mvneta_xdp_put_buff(pp, rxq, &xdp_buf, &sinfo, -1); + mvneta_xdp_put_buff(pp, rxq, &xdp_buf, -1); u64_stats_update_begin(&stats->syncp); stats->es.skb_alloc_error++; @@ -2457,11 +2446,10 @@ static int mvneta_rx_swbm(struct napi_struct *napi, napi_gro_receive(napi, skb); next: xdp_buf.data_hard_start = NULL; - sinfo.nr_frags = 0; } if (xdp_buf.data_hard_start) - mvneta_xdp_put_buff(pp, rxq, &xdp_buf, &sinfo, -1); + mvneta_xdp_put_buff(pp, rxq, &xdp_buf, -1); if (ps.xdp_redirect) xdp_do_flush_map(); From patchwork Wed Jul 28 09:38:11 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lorenzo Bianconi X-Patchwork-Id: 488474 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-19.7 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, INCLUDES_CR_TRAILER, INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B1C38C4320A for ; Wed, 28 Jul 2021 09:39:17 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 9F14160FC4 for ; Wed, 28 Jul 2021 09:39:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235679AbhG1JjS (ORCPT ); Wed, 28 Jul 2021 05:39:18 -0400 Received: from mail.kernel.org ([198.145.29.99]:51622 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235629AbhG1JjQ (ORCPT ); Wed, 28 Jul 2021 05:39:16 -0400 Received: by mail.kernel.org (Postfix) with ESMTPSA id 1EAF660F9C; Wed, 28 Jul 2021 09:39:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1627465155; bh=LjNCbLH+KltxB6FecSdPFA2T3RMpoqvnDB+kwWH5aw4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=BD7ct2Xx1OFE/i5AoveYlgLMWnYi6As/+lJkuz2YJQkgInfuQj3UrGZ8J/d5pr2ky 6RYPWSy7zPUlQhkBsIYafhzlMYbGRBrXeAeaidGtggeSjWbsGRy3SsrbrBjwzM3zk6 gT8pUvyplPUySvMDeiiDsXF6AnEIOwQBmNXjrZHCDmm1dR2KV1F1CJpZ0D/56arvsE tM+khfiDzfV3YgLlMaIX3vYlJeAYyUPX9nefnUElU0CBa6oYkIckiPhJbLUhkrqnqD /ulu8kzRXwm0x/Xg+/a8w+8Ke+vZYWg1QA7sKILjvdnKobwKoHhK0N4AF5MYMX1imC 5F9BTDLxusG6w== From: Lorenzo Bianconi To: bpf@vger.kernel.org, netdev@vger.kernel.org Cc: lorenzo.bianconi@redhat.com, davem@davemloft.net, kuba@kernel.org, ast@kernel.org, daniel@iogearbox.net, shayagr@amazon.com, john.fastabend@gmail.com, dsahern@kernel.org, brouer@redhat.com, echaudro@redhat.com, jasowang@redhat.com, alexander.duyck@gmail.com, saeed@kernel.org, maciej.fijalkowski@intel.com, magnus.karlsson@intel.com, tirthendu.sarkar@intel.com, toke@redhat.com Subject: [PATCH v10 bpf-next 06/18] net: marvell: rely on xdp_update_skb_shared_info utility routine Date: Wed, 28 Jul 2021 11:38:11 +0200 Message-Id: <44766448417b85de6364bde869f78345f212c849.1627463617.git.lorenzo@kernel.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org Rely on xdp_update_skb_shared_info routine in order to avoid resetting frags array in skb_shared_info structure building the skb in mvneta_swbm_build_skb(). Frags array is expected to be initialized by the receiving driver building the xdp_buff and here we just need to update memory metadata. Signed-off-by: Lorenzo Bianconi --- drivers/net/ethernet/marvell/mvneta.c | 44 +++++++++++++++------------ 1 file changed, 25 insertions(+), 19 deletions(-) diff --git a/drivers/net/ethernet/marvell/mvneta.c b/drivers/net/ethernet/marvell/mvneta.c index 60fc2d3ca619..c4489c848628 100644 --- a/drivers/net/ethernet/marvell/mvneta.c +++ b/drivers/net/ethernet/marvell/mvneta.c @@ -2304,11 +2304,24 @@ mvneta_swbm_add_rx_fragment(struct mvneta_port *pp, skb_frag_size_set(frag, data_len); __skb_frag_set_page(frag, page); - if (!xdp_buff_is_mb(xdp)) + /* We don't need to reset pp_recycle here. It's already set, so + * just mark fragments for recycling. + */ + page_pool_store_mem_info(page, rxq->page_pool); + + if (!xdp_buff_is_mb(xdp)) { + sinfo->xdp_frags_size = *size; xdp_buff_set_mb(xdp); + } + if (page_is_pfmemalloc(page)) + xdp_buff_set_frag_pfmemalloc(xdp); } else { page_pool_put_full_page(rxq->page_pool, page, true); } + + /* last fragment */ + if (len == *size) + sinfo->xdp_frags_tsize = sinfo->nr_frags * PAGE_SIZE; *size -= len; } @@ -2316,13 +2329,18 @@ static struct sk_buff * mvneta_swbm_build_skb(struct mvneta_port *pp, struct page_pool *pool, struct xdp_buff *xdp, u32 desc_status) { - struct skb_shared_info *sinfo = xdp_get_shared_info_from_buff(xdp); + unsigned int size, truesize; struct sk_buff *skb; u8 num_frags; - int i; - if (unlikely(xdp_buff_is_mb(xdp))) + if (unlikely(xdp_buff_is_mb(xdp))) { + struct skb_shared_info *sinfo; + + sinfo = xdp_get_shared_info_from_buff(xdp); + truesize = sinfo->xdp_frags_tsize; + size = sinfo->xdp_frags_size; num_frags = sinfo->nr_frags; + } skb = build_skb(xdp->data_hard_start, PAGE_SIZE); if (!skb) @@ -2334,22 +2352,10 @@ mvneta_swbm_build_skb(struct mvneta_port *pp, struct page_pool *pool, skb_put(skb, xdp->data_end - xdp->data); skb->ip_summed = mvneta_rx_csum(pp, desc_status); - if (likely(!xdp_buff_is_mb(xdp))) - goto out; - - for (i = 0; i < num_frags; i++) { - skb_frag_t *frag = &sinfo->frags[i]; - - skb_add_rx_frag(skb, skb_shinfo(skb)->nr_frags, - skb_frag_page(frag), skb_frag_off(frag), - skb_frag_size(frag), PAGE_SIZE); - /* We don't need to reset pp_recycle here. It's already set, so - * just mark fragments for recycling. - */ - page_pool_store_mem_info(skb_frag_page(frag), pool); - } + if (unlikely(xdp_buff_is_mb(xdp))) + xdp_update_skb_shared_info(skb, num_frags, size, truesize, + xdp_buff_is_frag_pfmemalloc(xdp)); -out: return skb; } From patchwork Wed Jul 28 09:38:13 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lorenzo Bianconi X-Patchwork-Id: 488473 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-19.7 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, INCLUDES_CR_TRAILER, INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1CE2FC4320A for ; Wed, 28 Jul 2021 09:39:26 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 0558A60FDA for ; Wed, 28 Jul 2021 09:39:26 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235724AbhG1Jj0 (ORCPT ); Wed, 28 Jul 2021 05:39:26 -0400 Received: from mail.kernel.org ([198.145.29.99]:51704 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231583AbhG1JjZ (ORCPT ); Wed, 28 Jul 2021 05:39:25 -0400 Received: by mail.kernel.org (Postfix) with ESMTPSA id 0FE2460FD8; Wed, 28 Jul 2021 09:39:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1627465164; bh=H7xnWnrxrHP7RfekM76MJRa5PiN1mqnv5HGycGqJjtQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=EWknnKhIf/XwiTEkZulKngVT4atrOMPH+F4VXsr2g264gSQI9uPKqELTOOZ+FLMf1 wz2degkYSfAngnvdGLdCp41cWA1SjTn0DiMhCC4iF73krCX1K1h1N/TxOcvoEQtSdJ utkch5t+d1xyeM23O4ogqFEoKjF+ZLjXwyXSnBIBMRkqAc+XAG9Nru4pPqgC2Nzi58 86Q1LskFhb73xAQ55vohg+LglLv9ctvjZKFORsjpoL+M6gT7QlPhMDrGUGfiz4b1jB nY1e2jxHBv1zxaYqXYS2Hru6dWrStLvaioGzZMb1uAdH2U7Sec/DruzV4UwMwXJF1r V1QP+1rA/+Wqg== From: Lorenzo Bianconi To: bpf@vger.kernel.org, netdev@vger.kernel.org Cc: lorenzo.bianconi@redhat.com, davem@davemloft.net, kuba@kernel.org, ast@kernel.org, daniel@iogearbox.net, shayagr@amazon.com, john.fastabend@gmail.com, dsahern@kernel.org, brouer@redhat.com, echaudro@redhat.com, jasowang@redhat.com, alexander.duyck@gmail.com, saeed@kernel.org, maciej.fijalkowski@intel.com, magnus.karlsson@intel.com, tirthendu.sarkar@intel.com, toke@redhat.com Subject: [PATCH v10 bpf-next 08/18] net: mvneta: add multi buffer support to XDP_TX Date: Wed, 28 Jul 2021 11:38:13 +0200 Message-Id: <8b9a36f984423e5da751a163bdc3666cf7fa5b86.1627463617.git.lorenzo@kernel.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org Introduce the capability to map non-linear xdp buffer running mvneta_xdp_submit_frame() for XDP_TX and XDP_REDIRECT Signed-off-by: Lorenzo Bianconi --- drivers/net/ethernet/marvell/mvneta.c | 112 +++++++++++++++++--------- 1 file changed, 76 insertions(+), 36 deletions(-) diff --git a/drivers/net/ethernet/marvell/mvneta.c b/drivers/net/ethernet/marvell/mvneta.c index c4489c848628..3abc75c80ec5 100644 --- a/drivers/net/ethernet/marvell/mvneta.c +++ b/drivers/net/ethernet/marvell/mvneta.c @@ -1856,8 +1856,8 @@ static void mvneta_txq_bufs_free(struct mvneta_port *pp, bytes_compl += buf->skb->len; pkts_compl++; dev_kfree_skb_any(buf->skb); - } else if (buf->type == MVNETA_TYPE_XDP_TX || - buf->type == MVNETA_TYPE_XDP_NDO) { + } else if ((buf->type == MVNETA_TYPE_XDP_TX || + buf->type == MVNETA_TYPE_XDP_NDO) && buf->xdpf) { if (napi && buf->type == MVNETA_TYPE_XDP_TX) xdp_return_frame_rx_napi(buf->xdpf); else @@ -2051,47 +2051,87 @@ mvneta_xdp_put_buff(struct mvneta_port *pp, struct mvneta_rx_queue *rxq, static int mvneta_xdp_submit_frame(struct mvneta_port *pp, struct mvneta_tx_queue *txq, - struct xdp_frame *xdpf, bool dma_map) + struct xdp_frame *xdpf, int *nxmit_byte, bool dma_map) { - struct mvneta_tx_desc *tx_desc; - struct mvneta_tx_buf *buf; - dma_addr_t dma_addr; + struct skb_shared_info *sinfo = xdp_get_shared_info_from_frame(xdpf); + struct device *dev = pp->dev->dev.parent; + struct mvneta_tx_desc *tx_desc = NULL; + int i, num_frames = 1; + struct page *page; + + if (unlikely(xdp_frame_is_mb(xdpf))) + num_frames += sinfo->nr_frags; - if (txq->count >= txq->tx_stop_threshold) + if (txq->count + num_frames >= txq->size) return MVNETA_XDP_DROPPED; - tx_desc = mvneta_txq_next_desc_get(txq); + for (i = 0; i < num_frames; i++) { + struct mvneta_tx_buf *buf = &txq->buf[txq->txq_put_index]; + skb_frag_t *frag = NULL; + int len = xdpf->len; + dma_addr_t dma_addr; - buf = &txq->buf[txq->txq_put_index]; - if (dma_map) { - /* ndo_xdp_xmit */ - dma_addr = dma_map_single(pp->dev->dev.parent, xdpf->data, - xdpf->len, DMA_TO_DEVICE); - if (dma_mapping_error(pp->dev->dev.parent, dma_addr)) { - mvneta_txq_desc_put(txq); - return MVNETA_XDP_DROPPED; + if (unlikely(i)) { /* paged area */ + frag = &sinfo->frags[i - 1]; + len = skb_frag_size(frag); } - buf->type = MVNETA_TYPE_XDP_NDO; - } else { - struct page *page = virt_to_page(xdpf->data); - dma_addr = page_pool_get_dma_addr(page) + - sizeof(*xdpf) + xdpf->headroom; - dma_sync_single_for_device(pp->dev->dev.parent, dma_addr, - xdpf->len, DMA_BIDIRECTIONAL); - buf->type = MVNETA_TYPE_XDP_TX; + tx_desc = mvneta_txq_next_desc_get(txq); + if (dma_map) { + /* ndo_xdp_xmit */ + void *data; + + data = unlikely(frag) ? skb_frag_address(frag) + : xdpf->data; + dma_addr = dma_map_single(dev, data, len, + DMA_TO_DEVICE); + if (dma_mapping_error(dev, dma_addr)) { + mvneta_txq_desc_put(txq); + goto unmap; + } + + buf->type = MVNETA_TYPE_XDP_NDO; + } else { + page = unlikely(frag) ? skb_frag_page(frag) + : virt_to_page(xdpf->data); + dma_addr = page_pool_get_dma_addr(page); + if (unlikely(frag)) + dma_addr += skb_frag_off(frag); + else + dma_addr += sizeof(*xdpf) + xdpf->headroom; + dma_sync_single_for_device(dev, dma_addr, len, + DMA_BIDIRECTIONAL); + buf->type = MVNETA_TYPE_XDP_TX; + } + buf->xdpf = unlikely(i) ? NULL : xdpf; + + tx_desc->command = unlikely(i) ? 0 : MVNETA_TXD_F_DESC; + tx_desc->buf_phys_addr = dma_addr; + tx_desc->data_size = len; + *nxmit_byte += len; + + mvneta_txq_inc_put(txq); } - buf->xdpf = xdpf; - tx_desc->command = MVNETA_TXD_FLZ_DESC; - tx_desc->buf_phys_addr = dma_addr; - tx_desc->data_size = xdpf->len; + /*last descriptor */ + if (likely(tx_desc)) + tx_desc->command |= MVNETA_TXD_L_DESC | MVNETA_TXD_Z_PAD; - mvneta_txq_inc_put(txq); - txq->pending++; - txq->count++; + txq->pending += num_frames; + txq->count += num_frames; return MVNETA_XDP_TX; + +unmap: + for (i--; i >= 0; i--) { + mvneta_txq_desc_put(txq); + tx_desc = txq->descs + txq->next_desc_to_proc; + dma_unmap_single(dev, tx_desc->buf_phys_addr, + tx_desc->data_size, + DMA_TO_DEVICE); + } + + return MVNETA_XDP_DROPPED; } static int @@ -2100,8 +2140,8 @@ mvneta_xdp_xmit_back(struct mvneta_port *pp, struct xdp_buff *xdp) struct mvneta_pcpu_stats *stats = this_cpu_ptr(pp->stats); struct mvneta_tx_queue *txq; struct netdev_queue *nq; + int cpu, nxmit_byte = 0; struct xdp_frame *xdpf; - int cpu; u32 ret; xdpf = xdp_convert_buff_to_frame(xdp); @@ -2113,10 +2153,10 @@ mvneta_xdp_xmit_back(struct mvneta_port *pp, struct xdp_buff *xdp) nq = netdev_get_tx_queue(pp->dev, txq->id); __netif_tx_lock(nq, cpu); - ret = mvneta_xdp_submit_frame(pp, txq, xdpf, false); + ret = mvneta_xdp_submit_frame(pp, txq, xdpf, &nxmit_byte, false); if (ret == MVNETA_XDP_TX) { u64_stats_update_begin(&stats->syncp); - stats->es.ps.tx_bytes += xdpf->len; + stats->es.ps.tx_bytes += nxmit_byte; stats->es.ps.tx_packets++; stats->es.ps.xdp_tx++; u64_stats_update_end(&stats->syncp); @@ -2155,11 +2195,11 @@ mvneta_xdp_xmit(struct net_device *dev, int num_frame, __netif_tx_lock(nq, cpu); for (i = 0; i < num_frame; i++) { - ret = mvneta_xdp_submit_frame(pp, txq, frames[i], true); + ret = mvneta_xdp_submit_frame(pp, txq, frames[i], &nxmit_byte, + true); if (ret != MVNETA_XDP_TX) break; - nxmit_byte += frames[i]->len; nxmit++; } From patchwork Wed Jul 28 09:38:15 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lorenzo Bianconi X-Patchwork-Id: 488472 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-19.7 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, INCLUDES_CR_TRAILER, INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A26C6C4320A for ; Wed, 28 Jul 2021 09:39:44 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 8B62260F9C for ; Wed, 28 Jul 2021 09:39:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235761AbhG1Jjo (ORCPT ); Wed, 28 Jul 2021 05:39:44 -0400 Received: from mail.kernel.org ([198.145.29.99]:51852 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235741AbhG1Jjn (ORCPT ); Wed, 28 Jul 2021 05:39:43 -0400 Received: by mail.kernel.org (Postfix) with ESMTPSA id 1E9D760FC0; Wed, 28 Jul 2021 09:39:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1627465182; bh=22tl80cnY03dTmXP3NY+ZjDnZlqEPVB3nsz6/KLvdt8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=TGGYO1XpZbWaSTts2Db4izoISvitdrBcUSCKDh6fNFZ0t464E7KCKWSFz6fnppsPb OGl7YYiveQeHBVo+78aKXeBCXEcgfTReoL6XqXcbYOPtJCcYEgaZfUETiE31KX2/uH J/sBgI9A8uO3dxhuVvzPQwOPLR5koo+Uhd4MnCXmfTr3UuGUb73+WoJVGkwA+qeZPw yR55Ja477b8L2gPdatzsQ3kilyW6qXZenwepwS2Zln9WnoZ38yite8iv4PJUdpXrKF BoenIianeFIyt7HMKP7+caEKCW5JXGLBCoh63OhJVoG+sCb3zgGGx5tpuVyry/mxAF Do/dy93RoKfUA== From: Lorenzo Bianconi To: bpf@vger.kernel.org, netdev@vger.kernel.org Cc: lorenzo.bianconi@redhat.com, davem@davemloft.net, kuba@kernel.org, ast@kernel.org, daniel@iogearbox.net, shayagr@amazon.com, john.fastabend@gmail.com, dsahern@kernel.org, brouer@redhat.com, echaudro@redhat.com, jasowang@redhat.com, alexander.duyck@gmail.com, saeed@kernel.org, maciej.fijalkowski@intel.com, magnus.karlsson@intel.com, tirthendu.sarkar@intel.com, toke@redhat.com Subject: [PATCH v10 bpf-next 10/18] bpf: add multi-buff support to the bpf_xdp_adjust_tail() API Date: Wed, 28 Jul 2021 11:38:15 +0200 Message-Id: X-Mailer: git-send-email 2.31.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Eelco Chaudron This change adds support for tail growing and shrinking for XDP multi-buff. When called on a multi-buffer packet with a grow request, it will always work on the last fragment of the packet. So the maximum grow size is the last fragments tailroom, i.e. no new buffer will be allocated. When shrinking, it will work from the last fragment, all the way down to the base buffer depending on the shrinking size. It's important to mention that once you shrink down the fragment(s) are freed, so you can not grow again to the original size. Co-developed-by: Lorenzo Bianconi Signed-off-by: Lorenzo Bianconi Signed-off-by: Eelco Chaudron --- include/net/xdp.h | 9 +++++++ net/core/filter.c | 60 +++++++++++++++++++++++++++++++++++++++++++++++ net/core/xdp.c | 5 ++-- 3 files changed, 72 insertions(+), 2 deletions(-) diff --git a/include/net/xdp.h b/include/net/xdp.h index d66e9877d773..cdaecf8d4d61 100644 --- a/include/net/xdp.h +++ b/include/net/xdp.h @@ -145,6 +145,13 @@ xdp_get_shared_info_from_buff(struct xdp_buff *xdp) return (struct skb_shared_info *)xdp_data_hard_end(xdp); } +static inline unsigned int xdp_get_frag_tailroom(const skb_frag_t *frag) +{ + struct page *page = skb_frag_page(frag); + + return page_size(page) - skb_frag_size(frag) - skb_frag_off(frag); +} + struct xdp_frame { void *data; u16 len; @@ -290,6 +297,8 @@ struct xdp_frame *xdp_convert_buff_to_frame(struct xdp_buff *xdp) return xdp_frame; } +void __xdp_return(void *data, struct xdp_mem_info *mem, bool napi_direct, + struct xdp_buff *xdp); void xdp_return_frame(struct xdp_frame *xdpf); void xdp_return_frame_rx_napi(struct xdp_frame *xdpf); void xdp_return_buff(struct xdp_buff *xdp); diff --git a/net/core/filter.c b/net/core/filter.c index faf29fd82276..e60e300b10cd 100644 --- a/net/core/filter.c +++ b/net/core/filter.c @@ -3835,11 +3835,71 @@ static const struct bpf_func_proto bpf_xdp_adjust_head_proto = { .arg2_type = ARG_ANYTHING, }; +static int bpf_xdp_mb_adjust_tail(struct xdp_buff *xdp, int offset) +{ + struct skb_shared_info *sinfo; + + sinfo = xdp_get_shared_info_from_buff(xdp); + if (offset >= 0) { + skb_frag_t *frag = &sinfo->frags[sinfo->nr_frags - 1]; + int size; + + if (unlikely(offset > xdp_get_frag_tailroom(frag))) + return -EINVAL; + + size = skb_frag_size(frag); + memset(skb_frag_address(frag) + size, 0, offset); + skb_frag_size_set(frag, size + offset); + sinfo->xdp_frags_size += offset; + } else { + int i, n_frags_free = 0, len_free = 0, tlen_free = 0; + + offset = abs(offset); + if (unlikely(offset > ((int)(xdp->data_end - xdp->data) + + sinfo->xdp_frags_size - ETH_HLEN))) + return -EINVAL; + + for (i = sinfo->nr_frags - 1; i >= 0 && offset > 0; i--) { + skb_frag_t *frag = &sinfo->frags[i]; + int size = skb_frag_size(frag); + int shrink = min_t(int, offset, size); + + len_free += shrink; + offset -= shrink; + + if (unlikely(size == shrink)) { + struct page *page = skb_frag_page(frag); + + __xdp_return(page_address(page), &xdp->rxq->mem, + false, NULL); + tlen_free += page_size(page); + n_frags_free++; + } else { + skb_frag_size_set(frag, size - shrink); + break; + } + } + sinfo->nr_frags -= n_frags_free; + sinfo->xdp_frags_size -= len_free; + sinfo->xdp_frags_tsize -= tlen_free; + + if (unlikely(offset > 0)) { + xdp_buff_clear_mb(xdp); + xdp->data_end -= offset; + } + } + + return 0; +} + BPF_CALL_2(bpf_xdp_adjust_tail, struct xdp_buff *, xdp, int, offset) { void *data_hard_end = xdp_data_hard_end(xdp); /* use xdp->frame_sz */ void *data_end = xdp->data_end + offset; + if (unlikely(xdp_buff_is_mb(xdp))) + return bpf_xdp_mb_adjust_tail(xdp, offset); + /* Notice that xdp_data_hard_end have reserved some tailroom */ if (unlikely(data_end > data_hard_end)) return -EINVAL; diff --git a/net/core/xdp.c b/net/core/xdp.c index 1346fb8b3f50..a71cdea75306 100644 --- a/net/core/xdp.c +++ b/net/core/xdp.c @@ -339,8 +339,8 @@ EXPORT_SYMBOL_GPL(xdp_rxq_info_reg_mem_model); * is used for those calls sites. Thus, allowing for faster recycling * of xdp_frames/pages in those cases. */ -static void __xdp_return(void *data, struct xdp_mem_info *mem, bool napi_direct, - struct xdp_buff *xdp) +void __xdp_return(void *data, struct xdp_mem_info *mem, bool napi_direct, + struct xdp_buff *xdp) { struct xdp_mem_allocator *xa; struct page *page; @@ -373,6 +373,7 @@ static void __xdp_return(void *data, struct xdp_mem_info *mem, bool napi_direct, break; } } +EXPORT_SYMBOL_GPL(__xdp_return); void xdp_return_frame(struct xdp_frame *xdpf) { From patchwork Wed Jul 28 09:38:17 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lorenzo Bianconi X-Patchwork-Id: 488471 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-19.7 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, INCLUDES_CR_TRAILER, INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6FE04C4320A for ; Wed, 28 Jul 2021 09:39:58 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 5342560F9E for ; Wed, 28 Jul 2021 09:39:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235787AbhG1Jj6 (ORCPT ); Wed, 28 Jul 2021 05:39:58 -0400 Received: from mail.kernel.org ([198.145.29.99]:51990 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235784AbhG1Jjv (ORCPT ); Wed, 28 Jul 2021 05:39:51 -0400 Received: by mail.kernel.org (Postfix) with ESMTPSA id 3A31B60F9C; Wed, 28 Jul 2021 09:39:47 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1627465190; bh=eULcNqTc93FqQ8ucsOGrtoQPPaH5YToZAvl63QOjeXg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=BF/vYhcMDyEodrPn6Hgb5YyCkJRFh4X4basON3kW7GC664A3U78XKTSR4ojJKZbdh n4YLIYf+L9q76yr+gxXcTG4RP2gerzILN+7SeJmTCqpJB56qzfgQLWzolX1fxwGF6J yYgFmgOAl/rYq4O+vy/+m7LJiNlJbaC5VQZB7OTT5JJyT6qp7iW286KGY8i3eUbGNs QmpkwI5KJjVjxH36vUxj1+vCuVSYkYjdho1x1udaa61G0kljvn6EDTT3TwPZHghvfc xQcycHERo4c5TqvnyTkC6Cm2fKFd+PAktkSmJvp+PNYvAd76av/b3t8FnFX3Pmd5d4 WligDUtuqw0qg== From: Lorenzo Bianconi To: bpf@vger.kernel.org, netdev@vger.kernel.org Cc: lorenzo.bianconi@redhat.com, davem@davemloft.net, kuba@kernel.org, ast@kernel.org, daniel@iogearbox.net, shayagr@amazon.com, john.fastabend@gmail.com, dsahern@kernel.org, brouer@redhat.com, echaudro@redhat.com, jasowang@redhat.com, alexander.duyck@gmail.com, saeed@kernel.org, maciej.fijalkowski@intel.com, magnus.karlsson@intel.com, tirthendu.sarkar@intel.com, toke@redhat.com Subject: [PATCH v10 bpf-next 12/18] bpf: add multi-buffer support to xdp copy helpers Date: Wed, 28 Jul 2021 11:38:17 +0200 Message-Id: <749f6d146902f7067343169162e91caca83ecef2.1627463617.git.lorenzo@kernel.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Eelco Chaudron This patch adds support for multi-buffer for the following helpers: - bpf_xdp_output() - bpf_perf_event_output() Signed-off-by: Eelco Chaudron Signed-off-by: Lorenzo Bianconi --- kernel/trace/bpf_trace.c | 3 + net/core/filter.c | 68 +++++++- .../selftests/bpf/prog_tests/xdp_bpf2bpf.c | 151 +++++++++++++----- .../selftests/bpf/progs/test_xdp_bpf2bpf.c | 2 +- 4 files changed, 180 insertions(+), 44 deletions(-) diff --git a/kernel/trace/bpf_trace.c b/kernel/trace/bpf_trace.c index c5e0b6a64091..054d42b6d080 100644 --- a/kernel/trace/bpf_trace.c +++ b/kernel/trace/bpf_trace.c @@ -1390,6 +1390,7 @@ static const struct bpf_func_proto bpf_perf_event_output_proto_raw_tp = { extern const struct bpf_func_proto bpf_skb_output_proto; extern const struct bpf_func_proto bpf_xdp_output_proto; +extern const struct bpf_func_proto bpf_xdp_get_buff_len_trace_proto; BPF_CALL_3(bpf_get_stackid_raw_tp, struct bpf_raw_tracepoint_args *, args, struct bpf_map *, map, u64, flags) @@ -1487,6 +1488,8 @@ tracing_prog_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog) return &bpf_sock_from_file_proto; case BPF_FUNC_get_socket_cookie: return &bpf_get_socket_ptr_cookie_proto; + case BPF_FUNC_xdp_get_buff_len: + return &bpf_xdp_get_buff_len_trace_proto; #endif case BPF_FUNC_seq_printf: return prog->expected_attach_type == BPF_TRACE_ITER ? diff --git a/net/core/filter.c b/net/core/filter.c index e3d34da19ef2..8f1a3b48cc6f 100644 --- a/net/core/filter.c +++ b/net/core/filter.c @@ -3822,6 +3822,15 @@ const struct bpf_func_proto bpf_xdp_get_buff_len_proto = { .arg1_type = ARG_PTR_TO_CTX, }; +BTF_ID_LIST_SINGLE(bpf_xdp_get_buff_len_bpf_ids, struct, xdp_buff) + +const struct bpf_func_proto bpf_xdp_get_buff_len_trace_proto = { + .func = bpf_xdp_get_buff_len, + .gpl_only = false, + .arg1_type = ARG_PTR_TO_BTF_ID, + .arg1_btf_id = &bpf_xdp_get_buff_len_bpf_ids[0], +}; + static unsigned long xdp_get_metalen(const struct xdp_buff *xdp) { return xdp_data_meta_unsupported(xdp) ? 0 : @@ -4611,10 +4620,52 @@ static const struct bpf_func_proto bpf_sk_ancestor_cgroup_id_proto = { }; #endif -static unsigned long bpf_xdp_copy(void *dst_buff, const void *src_buff, +static unsigned long bpf_xdp_copy(void *dst_buff, const void *ctx, unsigned long off, unsigned long len) { - memcpy(dst_buff, src_buff + off, len); + unsigned long base_len, copy_len, frag_off_total; + struct xdp_buff *xdp = (struct xdp_buff *)ctx; + struct skb_shared_info *sinfo; + int i; + + if (likely(!xdp_buff_is_mb(xdp))) { + memcpy(dst_buff, xdp->data + off, len); + return 0; + } + + base_len = xdp->data_end - xdp->data; + frag_off_total = base_len; + sinfo = xdp_get_shared_info_from_buff(xdp); + + /* If we need to copy data from the base buffer do it */ + if (off < base_len) { + copy_len = min(len, base_len - off); + memcpy(dst_buff, xdp->data + off, copy_len); + + off += copy_len; + len -= copy_len; + dst_buff += copy_len; + } + + /* Copy any remaining data from the fragments */ + for (i = 0; len && i < sinfo->nr_frags; i++) { + skb_frag_t *frag = &sinfo->frags[i]; + unsigned long frag_len, frag_off; + + frag_len = skb_frag_size(frag); + frag_off = off - frag_off_total; + if (frag_off < frag_len) { + copy_len = min(len, frag_len - frag_off); + memcpy(dst_buff, + skb_frag_address(frag) + frag_off, copy_len); + + off += copy_len; + len -= copy_len; + dst_buff += copy_len; + } + frag_off_total += frag_len; + } + return 0; } @@ -4626,10 +4677,19 @@ BPF_CALL_5(bpf_xdp_event_output, struct xdp_buff *, xdp, struct bpf_map *, map, if (unlikely(flags & ~(BPF_F_CTXLEN_MASK | BPF_F_INDEX_MASK))) return -EINVAL; if (unlikely(!xdp || - xdp_size > (unsigned long)(xdp->data_end - xdp->data))) + (likely(!xdp_buff_is_mb(xdp)) && + xdp_size > (unsigned long)(xdp->data_end - xdp->data)))) return -EFAULT; + if (unlikely(xdp_buff_is_mb(xdp))) { + struct skb_shared_info *sinfo; + + sinfo = xdp_get_shared_info_from_buff(xdp); + if (unlikely(xdp_size > ((int)(xdp->data_end - xdp->data) + + sinfo->xdp_frags_size))) + return -EFAULT; + } - return bpf_event_output(map, flags, meta, meta_size, xdp->data, + return bpf_event_output(map, flags, meta, meta_size, xdp, xdp_size, bpf_xdp_copy); } diff --git a/tools/testing/selftests/bpf/prog_tests/xdp_bpf2bpf.c b/tools/testing/selftests/bpf/prog_tests/xdp_bpf2bpf.c index 3bd5904b4db5..fe279c1c0e48 100644 --- a/tools/testing/selftests/bpf/prog_tests/xdp_bpf2bpf.c +++ b/tools/testing/selftests/bpf/prog_tests/xdp_bpf2bpf.c @@ -10,11 +10,20 @@ struct meta { int pkt_len; }; +struct test_ctx_s { + bool passed; + int pkt_size; +}; + +struct test_ctx_s test_ctx; + static void on_sample(void *ctx, int cpu, void *data, __u32 size) { - int duration = 0; struct meta *meta = (struct meta *)data; struct ipv4_packet *trace_pkt_v4 = data + sizeof(*meta); + unsigned char *raw_pkt = data + sizeof(*meta); + struct test_ctx_s *tst_ctx = ctx; + int duration = 0; if (CHECK(size < sizeof(pkt_v4) + sizeof(*meta), "check_size", "size %u < %zu\n", @@ -25,25 +34,114 @@ static void on_sample(void *ctx, int cpu, void *data, __u32 size) "meta->ifindex = %d\n", meta->ifindex)) return; - if (CHECK(meta->pkt_len != sizeof(pkt_v4), "check_meta_pkt_len", - "meta->pkt_len = %zd\n", sizeof(pkt_v4))) + if (CHECK(meta->pkt_len != tst_ctx->pkt_size, "check_meta_pkt_len", + "meta->pkt_len = %d\n", tst_ctx->pkt_size)) return; if (CHECK(memcmp(trace_pkt_v4, &pkt_v4, sizeof(pkt_v4)), "check_packet_content", "content not the same\n")) return; - *(bool *)ctx = true; + if (meta->pkt_len > sizeof(pkt_v4)) { + for (int i = 0; i < (meta->pkt_len - sizeof(pkt_v4)); i++) { + if (raw_pkt[i + sizeof(pkt_v4)] != (unsigned char)i) { + CHECK(true, "check_packet_content", + "byte %zu does not match %u != %u\n", + i + sizeof(pkt_v4), + raw_pkt[i + sizeof(pkt_v4)], + (unsigned char)i); + break; + } + } + } + + tst_ctx->passed = true; } -void test_xdp_bpf2bpf(void) +#define BUF_SZ 9000 + +static int run_xdp_bpf2bpf_pkt_size(int pkt_fd, struct perf_buffer *pb, + struct test_xdp_bpf2bpf *ftrace_skel, + int pkt_size) { __u32 duration = 0, retval, size; - char buf[128]; + __u8 *buf, *buf_in; + int err, ret = 0; + + if (pkt_size > BUF_SZ || pkt_size < sizeof(pkt_v4)) + return -EINVAL; + + buf_in = malloc(BUF_SZ); + if (CHECK(!buf_in, "buf_in malloc()", "error:%s\n", strerror(errno))) + return -ENOMEM; + + buf = malloc(BUF_SZ); + if (CHECK(!buf, "buf malloc()", "error:%s\n", strerror(errno))) { + ret = -ENOMEM; + goto free_buf_in; + } + + test_ctx.passed = false; + test_ctx.pkt_size = pkt_size; + + memcpy(buf_in, &pkt_v4, sizeof(pkt_v4)); + if (pkt_size > sizeof(pkt_v4)) { + for (int i = 0; i < (pkt_size - sizeof(pkt_v4)); i++) + buf_in[i + sizeof(pkt_v4)] = i; + } + + /* Run test program */ + err = bpf_prog_test_run(pkt_fd, 1, buf_in, pkt_size, + buf, &size, &retval, &duration); + + if (CHECK(err || retval != XDP_PASS || size != pkt_size, + "ipv4", "err %d errno %d retval %d size %d\n", + err, errno, retval, size)) { + ret = err ? err : -EINVAL; + goto free_buf; + } + + /* Make sure bpf_xdp_output() was triggered and it sent the expected + * data to the perf ring buffer. + */ + err = perf_buffer__poll(pb, 100); + if (CHECK(err <= 0, "perf_buffer__poll", "err %d\n", err)) { + ret = -EINVAL; + goto free_buf; + } + + if (CHECK_FAIL(!test_ctx.passed)) { + ret = -EINVAL; + goto free_buf; + } + + /* Verify test results */ + if (CHECK(ftrace_skel->bss->test_result_fentry != if_nametoindex("lo"), + "result", "fentry failed err %llu\n", + ftrace_skel->bss->test_result_fentry)) { + ret = -EINVAL; + goto free_buf; + } + + if (CHECK(ftrace_skel->bss->test_result_fexit != XDP_PASS, "result", + "fexit failed err %llu\n", + ftrace_skel->bss->test_result_fexit)) + ret = -EINVAL; + +free_buf: + free(buf); +free_buf_in: + free(buf_in); + + return ret; +} + +void test_xdp_bpf2bpf(void) +{ int err, pkt_fd, map_fd; - bool passed = false; - struct iphdr *iph = (void *)buf + sizeof(struct ethhdr); - struct iptnl_info value4 = {.family = AF_INET}; + __u32 duration = 0; + int pkt_sizes[] = {sizeof(pkt_v4), 1024, 4100, 8200}; + struct iptnl_info value4 = {.family = AF_INET6}; struct test_xdp *pkt_skel = NULL; struct test_xdp_bpf2bpf *ftrace_skel = NULL; struct vip key4 = {.protocol = 6, .family = AF_INET}; @@ -87,40 +185,15 @@ void test_xdp_bpf2bpf(void) /* Set up perf buffer */ pb_opts.sample_cb = on_sample; - pb_opts.ctx = &passed; + pb_opts.ctx = &test_ctx; pb = perf_buffer__new(bpf_map__fd(ftrace_skel->maps.perf_buf_map), - 1, &pb_opts); + 8, &pb_opts); if (!ASSERT_OK_PTR(pb, "perf_buf__new")) goto out; - /* Run test program */ - err = bpf_prog_test_run(pkt_fd, 1, &pkt_v4, sizeof(pkt_v4), - buf, &size, &retval, &duration); - - if (CHECK(err || retval != XDP_TX || size != 74 || - iph->protocol != IPPROTO_IPIP, "ipv4", - "err %d errno %d retval %d size %d\n", - err, errno, retval, size)) - goto out; - - /* Make sure bpf_xdp_output() was triggered and it sent the expected - * data to the perf ring buffer. - */ - err = perf_buffer__poll(pb, 100); - if (CHECK(err < 0, "perf_buffer__poll", "err %d\n", err)) - goto out; - - CHECK_FAIL(!passed); - - /* Verify test results */ - if (CHECK(ftrace_skel->bss->test_result_fentry != if_nametoindex("lo"), - "result", "fentry failed err %llu\n", - ftrace_skel->bss->test_result_fentry)) - goto out; - - CHECK(ftrace_skel->bss->test_result_fexit != XDP_TX, "result", - "fexit failed err %llu\n", ftrace_skel->bss->test_result_fexit); - + for (int i = 0; i < ARRAY_SIZE(pkt_sizes); i++) + run_xdp_bpf2bpf_pkt_size(pkt_fd, pb, ftrace_skel, + pkt_sizes[i]); out: if (pb) perf_buffer__free(pb); diff --git a/tools/testing/selftests/bpf/progs/test_xdp_bpf2bpf.c b/tools/testing/selftests/bpf/progs/test_xdp_bpf2bpf.c index a038e827f850..902b54190377 100644 --- a/tools/testing/selftests/bpf/progs/test_xdp_bpf2bpf.c +++ b/tools/testing/selftests/bpf/progs/test_xdp_bpf2bpf.c @@ -49,7 +49,7 @@ int BPF_PROG(trace_on_entry, struct xdp_buff *xdp) void *data = (void *)(long)xdp->data; meta.ifindex = xdp->rxq->dev->ifindex; - meta.pkt_len = data_end - data; + meta.pkt_len = bpf_xdp_get_buff_len((struct xdp_md *)xdp); bpf_xdp_output(xdp, &perf_buf_map, ((__u64) meta.pkt_len << 32) | BPF_F_CURRENT_CPU, From patchwork Wed Jul 28 09:38:19 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lorenzo Bianconi X-Patchwork-Id: 488470 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-19.7 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, INCLUDES_CR_TRAILER, INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 166F0C432BE for ; Wed, 28 Jul 2021 09:40:04 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id EE68360F9E for ; Wed, 28 Jul 2021 09:40:03 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235797AbhG1JkE (ORCPT ); Wed, 28 Jul 2021 05:40:04 -0400 Received: from mail.kernel.org ([198.145.29.99]:52094 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235785AbhG1Jj6 (ORCPT ); Wed, 28 Jul 2021 05:39:58 -0400 Received: by mail.kernel.org (Postfix) with ESMTPSA id 4EEC360FC4; Wed, 28 Jul 2021 09:39:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1627465197; bh=6wI9QopYFGN/7pouTeoe8Ek4JZPQdQ4Pa3lrnggRabA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=WQckAmIAkkjWZMaHW51rCVPNAq9iJ3/LWjL6mwKCBA6nRPmpbZPp+YE76gA1Hu+tF xd3aXPTpWDjy+6sJER+2xLKRLnnZPlX3/Dc0f4Ay7/RLl0YyLRCY/B1iZpldrjj6HS 3ze855uJdh6FtRghrmG2NuaAixs6TlkS54fZczop/83g2g8O7ZjwC0kcoo4osQ2Jro ff+pY/kwyljoBddcQIRJGkpYnCmFgdOeJ+qku+bQtfPSy6MJJIOAZ+PPkmA+yZYwzb wVXYAdohNCvL6VrDFMRbrkKLXjWeOV4aBXCqovHBrU0KKWH8S0YMua9B90TBKvp9HK IPRDnXiU0OwdA== From: Lorenzo Bianconi To: bpf@vger.kernel.org, netdev@vger.kernel.org Cc: lorenzo.bianconi@redhat.com, davem@davemloft.net, kuba@kernel.org, ast@kernel.org, daniel@iogearbox.net, shayagr@amazon.com, john.fastabend@gmail.com, dsahern@kernel.org, brouer@redhat.com, echaudro@redhat.com, jasowang@redhat.com, alexander.duyck@gmail.com, saeed@kernel.org, maciej.fijalkowski@intel.com, magnus.karlsson@intel.com, tirthendu.sarkar@intel.com, toke@redhat.com Subject: [PATCH v10 bpf-next 14/18] bpf: introduce multibuff support to bpf_prog_test_run_xdp() Date: Wed, 28 Jul 2021 11:38:19 +0200 Message-Id: X-Mailer: git-send-email 2.31.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org Introduce the capability to allocate a xdp multi-buff in bpf_prog_test_run_xdp routine. This is a preliminary patch to introduce the selftests for new xdp multi-buff ebpf helpers Signed-off-by: Lorenzo Bianconi --- net/bpf/test_run.c | 54 ++++++++++++++++++++++++++++++++++++---------- 1 file changed, 43 insertions(+), 11 deletions(-) diff --git a/net/bpf/test_run.c b/net/bpf/test_run.c index f514bfe9837b..529c9d9ae702 100644 --- a/net/bpf/test_run.c +++ b/net/bpf/test_run.c @@ -749,16 +749,16 @@ int bpf_prog_test_run_xdp(struct bpf_prog *prog, const union bpf_attr *kattr, union bpf_attr __user *uattr) { u32 tailroom = SKB_DATA_ALIGN(sizeof(struct skb_shared_info)); - u32 headroom = XDP_PACKET_HEADROOM; u32 size = kattr->test.data_size_in; + u32 headroom = XDP_PACKET_HEADROOM; + u32 retval, duration, max_data_sz; u32 repeat = kattr->test.repeat; struct netdev_rx_queue *rxqueue; + struct skb_shared_info *sinfo; struct xdp_buff xdp = {}; - u32 retval, duration; + int i, ret = -EINVAL; struct xdp_md *ctx; - u32 max_data_sz; void *data; - int ret = -EINVAL; ctx = bpf_ctx_init(kattr, sizeof(struct xdp_md)); if (IS_ERR(ctx)) @@ -774,11 +774,10 @@ int bpf_prog_test_run_xdp(struct bpf_prog *prog, const union bpf_attr *kattr, headroom -= ctx->data; } - /* XDP have extra tailroom as (most) drivers use full page */ max_data_sz = 4096 - headroom - tailroom; + size = min_t(u32, size, max_data_sz); - data = bpf_test_init(kattr, kattr->test.data_size_in, - max_data_sz, headroom, tailroom); + data = bpf_test_init(kattr, size, max_data_sz, headroom, tailroom); if (IS_ERR(data)) { ret = PTR_ERR(data); goto free_ctx; @@ -788,11 +787,45 @@ int bpf_prog_test_run_xdp(struct bpf_prog *prog, const union bpf_attr *kattr, xdp_init_buff(&xdp, headroom + max_data_sz + tailroom, &rxqueue->xdp_rxq); xdp_prepare_buff(&xdp, data, headroom, size, true); + sinfo = xdp_get_shared_info_from_buff(&xdp); ret = xdp_convert_md_to_buff(ctx, &xdp); if (ret) goto free_data; + if (unlikely(kattr->test.data_size_in > size)) { + void __user *data_in = u64_to_user_ptr(kattr->test.data_in); + + while (size < kattr->test.data_size_in) { + struct page *page; + skb_frag_t *frag; + int data_len; + + page = alloc_page(GFP_KERNEL); + if (!page) { + ret = -ENOMEM; + goto out; + } + + frag = &sinfo->frags[sinfo->nr_frags++]; + __skb_frag_set_page(frag, page); + + data_len = min_t(int, kattr->test.data_size_in - size, + PAGE_SIZE); + skb_frag_size_set(frag, data_len); + + if (copy_from_user(page_address(page), data_in + size, + data_len)) { + ret = -EFAULT; + goto out; + } + sinfo->xdp_frags_tsize += PAGE_SIZE; + sinfo->xdp_frags_size += data_len; + size += data_len; + } + xdp_buff_set_mb(&xdp); + } + bpf_prog_change_xdp(NULL, prog); ret = bpf_test_run(prog, &xdp, repeat, &retval, &duration, true); /* We convert the xdp_buff back to an xdp_md before checking the return @@ -803,10 +836,7 @@ int bpf_prog_test_run_xdp(struct bpf_prog *prog, const union bpf_attr *kattr, if (ret) goto out; - if (xdp.data_meta != data + headroom || - xdp.data_end != xdp.data_meta + size) - size = xdp.data_end - xdp.data_meta; - + size = xdp.data_end - xdp.data_meta + sinfo->xdp_frags_size; ret = bpf_test_finish(kattr, uattr, xdp.data_meta, size, retval, duration); if (!ret) @@ -816,6 +846,8 @@ int bpf_prog_test_run_xdp(struct bpf_prog *prog, const union bpf_attr *kattr, out: bpf_prog_change_xdp(prog, NULL); free_data: + for (i = 0; i < sinfo->nr_frags; i++) + __free_page(skb_frag_page(&sinfo->frags[i])); kfree(data); free_ctx: kfree(ctx); From patchwork Wed Jul 28 09:38:21 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lorenzo Bianconi X-Patchwork-Id: 488469 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-19.7 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, INCLUDES_CR_TRAILER, INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1FF71C4320A for ; Wed, 28 Jul 2021 09:40:10 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 0CF3360F9E for ; Wed, 28 Jul 2021 09:40:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235837AbhG1JkK (ORCPT ); Wed, 28 Jul 2021 05:40:10 -0400 Received: from mail.kernel.org ([198.145.29.99]:52244 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235549AbhG1JkG (ORCPT ); Wed, 28 Jul 2021 05:40:06 -0400 Received: by mail.kernel.org (Postfix) with ESMTPSA id 5A56B60FC0; Wed, 28 Jul 2021 09:40:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1627465204; bh=+2GkSh7OgF7AftQVuWyu4GKMtX7I9oWEYjiLcT2xLDY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=nrTcz3lQvFSySJpjv1m6t8rYP0EpLJSNHEY2Ac2HipYa7CxlMwLI6We1wRBkDgkYp REY1f3tli4G0wOkVv2AHY9SAAE/Aj/fNvcU9mfTGcHQqU0YVoDhF+n1jy+5xoP8Uq1 3PBD2mMvArzB5ACISOWHf929KXZPSSQIFB4Sq6E2n7FN1VukEhVBIouQrYpgPfugnI nxMQHYgopj1xP7CqV7l2ed/3cKpQ0IQnnKb0zo7suagslttM5m2CPq/V4QoEtC0fiN OLEoYeHwa4gT/+htICr2Oamhfw3txcbk1xdQm3K8RpzP7+5HlA/Wgx4Wv4cywcs5VL IYCo+1x6KH+FQ== From: Lorenzo Bianconi To: bpf@vger.kernel.org, netdev@vger.kernel.org Cc: lorenzo.bianconi@redhat.com, davem@davemloft.net, kuba@kernel.org, ast@kernel.org, daniel@iogearbox.net, shayagr@amazon.com, john.fastabend@gmail.com, dsahern@kernel.org, brouer@redhat.com, echaudro@redhat.com, jasowang@redhat.com, alexander.duyck@gmail.com, saeed@kernel.org, maciej.fijalkowski@intel.com, magnus.karlsson@intel.com, tirthendu.sarkar@intel.com, toke@redhat.com Subject: [PATCH v10 bpf-next 16/18] bpf: update xdp_adjust_tail selftest to include multi-buffer Date: Wed, 28 Jul 2021 11:38:21 +0200 Message-Id: <40f6bd3a7929fc9f9bf03cdb7b7acecce4c60a62.1627463617.git.lorenzo@kernel.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Eelco Chaudron This change adds test cases for the multi-buffer scenarios when shrinking and growing. Signed-off-by: Eelco Chaudron Signed-off-by: Lorenzo Bianconi --- .../bpf/prog_tests/xdp_adjust_tail.c | 118 ++++++++++++++++++ .../bpf/progs/test_xdp_adjust_tail_grow.c | 10 +- .../bpf/progs/test_xdp_adjust_tail_shrink.c | 32 ++++- 3 files changed, 153 insertions(+), 7 deletions(-) diff --git a/tools/testing/selftests/bpf/prog_tests/xdp_adjust_tail.c b/tools/testing/selftests/bpf/prog_tests/xdp_adjust_tail.c index d5c98f2cb12f..40f7ae798fd1 100644 --- a/tools/testing/selftests/bpf/prog_tests/xdp_adjust_tail.c +++ b/tools/testing/selftests/bpf/prog_tests/xdp_adjust_tail.c @@ -130,6 +130,120 @@ void test_xdp_adjust_tail_grow2(void) bpf_object__close(obj); } +void test_xdp_adjust_mb_tail_shrink(void) +{ + const char *file = "./test_xdp_adjust_tail_shrink.o"; + __u32 duration, retval, size, exp_size; + struct bpf_object *obj; + int err, prog_fd; + __u8 *buf; + + /* For the individual test cases, the first byte in the packet + * indicates which test will be run. + */ + + err = bpf_prog_load(file, BPF_PROG_TYPE_XDP, &obj, &prog_fd); + if (CHECK_FAIL(err)) + return; + + buf = malloc(9000); + if (CHECK(!buf, "malloc()", "error:%s\n", strerror(errno))) + return; + + memset(buf, 0, 9000); + + /* Test case removing 10 bytes from last frag, NOT freeing it */ + exp_size = 8990; /* 9000 - 10 */ + err = bpf_prog_test_run(prog_fd, 1, buf, 9000, + buf, &size, &retval, &duration); + + CHECK(err || retval != XDP_TX || size != exp_size, + "9k-10b", "err %d errno %d retval %d[%d] size %d[%u]\n", + err, errno, retval, XDP_TX, size, exp_size); + + /* Test case removing one of two pages, assuming 4K pages */ + buf[0] = 1; + exp_size = 4900; /* 9000 - 4100 */ + err = bpf_prog_test_run(prog_fd, 1, buf, 9000, + buf, &size, &retval, &duration); + + CHECK(err || retval != XDP_TX || size != exp_size, + "9k-1p", "err %d errno %d retval %d[%d] size %d[%u]\n", + err, errno, retval, XDP_TX, size, exp_size); + + /* Test case removing two pages resulting in a non mb xdp_buff */ + buf[0] = 2; + exp_size = 800; /* 9000 - 8200 */ + err = bpf_prog_test_run(prog_fd, 1, buf, 9000, + buf, &size, &retval, &duration); + + CHECK(err || retval != XDP_TX || size != exp_size, + "9k-2p", "err %d errno %d retval %d[%d] size %d[%u]\n", + err, errno, retval, XDP_TX, size, exp_size); + + free(buf); + + bpf_object__close(obj); +} + +void test_xdp_adjust_mb_tail_grow(void) +{ + const char *file = "./test_xdp_adjust_tail_grow.o"; + __u32 duration, retval, size, exp_size; + struct bpf_object *obj; + int err, i, prog_fd; + __u8 *buf; + + err = bpf_prog_load(file, BPF_PROG_TYPE_XDP, &obj, &prog_fd); + if (CHECK_FAIL(err)) + return; + + buf = malloc(16384); + if (CHECK(!buf, "malloc()", "error:%s\n", strerror(errno))) + return; + + /* Test case add 10 bytes to last frag */ + memset(buf, 1, 16384); + size = 9000; + exp_size = size + 10; + err = bpf_prog_test_run(prog_fd, 1, buf, size, + buf, &size, &retval, &duration); + + CHECK(err || retval != XDP_TX || size != exp_size, + "9k+10b", "err %d retval %d[%d] size %d[%u]\n", + err, retval, XDP_TX, size, exp_size); + + for (i = 0; i < 9000; i++) + CHECK(buf[i] != 1, "9k+10b-old", + "Old data not all ok, offset %i is failing [%u]!\n", + i, buf[i]); + + for (i = 9000; i < 9010; i++) + CHECK(buf[i] != 0, "9k+10b-new", + "New data not all ok, offset %i is failing [%u]!\n", + i, buf[i]); + + for (i = 9010; i < 16384; i++) + CHECK(buf[i] != 1, "9k+10b-untouched", + "Unused data not all ok, offset %i is failing [%u]!\n", + i, buf[i]); + + /* Test a too large grow */ + memset(buf, 1, 16384); + size = 9001; + exp_size = size; + err = bpf_prog_test_run(prog_fd, 1, buf, size, + buf, &size, &retval, &duration); + + CHECK(err || retval != XDP_DROP || size != exp_size, + "9k+10b", "err %d retval %d[%d] size %d[%u]\n", + err, retval, XDP_TX, size, exp_size); + + free(buf); + + bpf_object__close(obj); +} + void test_xdp_adjust_tail(void) { if (test__start_subtest("xdp_adjust_tail_shrink")) @@ -138,4 +252,8 @@ void test_xdp_adjust_tail(void) test_xdp_adjust_tail_grow(); if (test__start_subtest("xdp_adjust_tail_grow2")) test_xdp_adjust_tail_grow2(); + if (test__start_subtest("xdp_adjust_mb_tail_shrink")) + test_xdp_adjust_mb_tail_shrink(); + if (test__start_subtest("xdp_adjust_mb_tail_grow")) + test_xdp_adjust_mb_tail_grow(); } diff --git a/tools/testing/selftests/bpf/progs/test_xdp_adjust_tail_grow.c b/tools/testing/selftests/bpf/progs/test_xdp_adjust_tail_grow.c index 3d66599eee2e..3d43defb0e00 100644 --- a/tools/testing/selftests/bpf/progs/test_xdp_adjust_tail_grow.c +++ b/tools/testing/selftests/bpf/progs/test_xdp_adjust_tail_grow.c @@ -7,11 +7,10 @@ int _xdp_adjust_tail_grow(struct xdp_md *xdp) { void *data_end = (void *)(long)xdp->data_end; void *data = (void *)(long)xdp->data; - unsigned int data_len; + int data_len = bpf_xdp_get_buff_len(xdp); int offset = 0; /* Data length determine test case */ - data_len = data_end - data; if (data_len == 54) { /* sizeof(pkt_v4) */ offset = 4096; /* test too large offset */ @@ -20,7 +19,12 @@ int _xdp_adjust_tail_grow(struct xdp_md *xdp) } else if (data_len == 64) { offset = 128; } else if (data_len == 128) { - offset = 4096 - 256 - 320 - data_len; /* Max tail grow 3520 */ + /* Max tail grow 3520 */ + offset = 4096 - 256 - 320 - data_len; + } else if (data_len == 9000) { + offset = 10; + } else if (data_len == 9001) { + offset = 4096; } else { return XDP_ABORTED; /* No matching test */ } diff --git a/tools/testing/selftests/bpf/progs/test_xdp_adjust_tail_shrink.c b/tools/testing/selftests/bpf/progs/test_xdp_adjust_tail_shrink.c index 22065a9cfb25..64177597ac29 100644 --- a/tools/testing/selftests/bpf/progs/test_xdp_adjust_tail_shrink.c +++ b/tools/testing/selftests/bpf/progs/test_xdp_adjust_tail_shrink.c @@ -14,14 +14,38 @@ int _version SEC("version") = 1; SEC("xdp_adjust_tail_shrink") int _xdp_adjust_tail_shrink(struct xdp_md *xdp) { - void *data_end = (void *)(long)xdp->data_end; - void *data = (void *)(long)xdp->data; + __u8 *data_end = (void *)(long)xdp->data_end; + __u8 *data = (void *)(long)xdp->data; int offset = 0; - if (data_end - data == 54) /* sizeof(pkt_v4) */ + switch (bpf_xdp_get_buff_len(xdp)) { + case 54: + /* sizeof(pkt_v4) */ offset = 256; /* shrink too much */ - else + break; + case 9000: + /* Multi-buffer test cases */ + if (data + 1 > data_end) + return XDP_DROP; + + switch (data[0]) { + case 0: + offset = 10; + break; + case 1: + offset = 4100; + break; + case 2: + offset = 8200; + break; + default: + return XDP_DROP; + } + break; + default: offset = 20; + break; + } if (bpf_xdp_adjust_tail(xdp, 0 - offset)) return XDP_DROP; return XDP_TX; From patchwork Wed Jul 28 09:38:23 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lorenzo Bianconi X-Patchwork-Id: 488468 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-19.7 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, INCLUDES_CR_TRAILER, INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 25D59C4320A for ; Wed, 28 Jul 2021 09:40:18 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 1273460F9C for ; Wed, 28 Jul 2021 09:40:18 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235836AbhG1JkR (ORCPT ); Wed, 28 Jul 2021 05:40:17 -0400 Received: from mail.kernel.org ([198.145.29.99]:52326 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231524AbhG1JkM (ORCPT ); Wed, 28 Jul 2021 05:40:12 -0400 Received: by mail.kernel.org (Postfix) with ESMTPSA id 6C5A560FE7; Wed, 28 Jul 2021 09:40:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1627465211; bh=L6HltSPqAJA403xBR4Z2bmI0frjkR0Z1YCQaSySnqII=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Z/6laCG/xgemgCoPf05csZBNNoiaLhgcdLO3uYHVZvTQ8JTiDyBpAOFksjcxxKWt4 lQ2WnZoxWCEN4zrJHIAUQIF83j96USyFlx4DNC/KYAl28TIRTYjLlC9YJjpH/tQdlv BOv8TvilhTPpqXkUyYE9MYqeqDT/VCL/UdednIJBEirrBMokgV7chBvZ4D7VdrTBb1 xT3xVaMTRO3PNXn2XnBcMmvMtoYprl4Oc+H1INApVQW4n0h2DTwHp6+UaEgKPXxZKr m9aOVIO0zEKRjRWQjm/FVBA5lN8x6cMFtiYXogsGdhwiLPbkYInbep1tZBa9t/SAUn jTsRO84iem+AA== From: Lorenzo Bianconi To: bpf@vger.kernel.org, netdev@vger.kernel.org Cc: lorenzo.bianconi@redhat.com, davem@davemloft.net, kuba@kernel.org, ast@kernel.org, daniel@iogearbox.net, shayagr@amazon.com, john.fastabend@gmail.com, dsahern@kernel.org, brouer@redhat.com, echaudro@redhat.com, jasowang@redhat.com, alexander.duyck@gmail.com, saeed@kernel.org, maciej.fijalkowski@intel.com, magnus.karlsson@intel.com, tirthendu.sarkar@intel.com, toke@redhat.com Subject: [PATCH v10 bpf-next 18/18] bpf: add bpf_xdp_adjust_data selftest Date: Wed, 28 Jul 2021 11:38:23 +0200 Message-Id: <8761f8ba2fd7f2940017c5b4c2aac1c6beeed066.1627463617.git.lorenzo@kernel.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org Introduce kernel selftest for new bpf_xdp_adjust_data helper. Signed-off-by: Lorenzo Bianconi --- .../bpf/prog_tests/xdp_adjust_data.c | 55 +++++++++++++++++++ .../bpf/progs/test_xdp_update_frags.c | 49 +++++++++++++++++ 2 files changed, 104 insertions(+) create mode 100644 tools/testing/selftests/bpf/prog_tests/xdp_adjust_data.c create mode 100644 tools/testing/selftests/bpf/progs/test_xdp_update_frags.c diff --git a/tools/testing/selftests/bpf/prog_tests/xdp_adjust_data.c b/tools/testing/selftests/bpf/prog_tests/xdp_adjust_data.c new file mode 100644 index 000000000000..a3e098b72fc9 --- /dev/null +++ b/tools/testing/selftests/bpf/prog_tests/xdp_adjust_data.c @@ -0,0 +1,55 @@ +// SPDX-License-Identifier: GPL-2.0 +#include +#include + +void test_xdp_update_frag(void) +{ + const char *file = "./test_xdp_update_frags.o"; + __u32 duration, retval, size; + struct bpf_object *obj; + int err, prog_fd; + __u8 *buf; + + err = bpf_prog_load(file, BPF_PROG_TYPE_XDP, &obj, &prog_fd); + if (CHECK_FAIL(err)) + return; + + buf = malloc(128); + if (CHECK(!buf, "malloc()", "error:%s\n", strerror(errno))) + return; + + memset(buf, 0, 128); + + err = bpf_prog_test_run(prog_fd, 1, buf, 128, + buf, &size, &retval, &duration); + free(buf); + + CHECK(err || retval != XDP_DROP, + "128b", "err %d errno %d retval %d size %d\n", + err, errno, retval, size); + + buf = malloc(9000); + if (CHECK(!buf, "malloc()", "error:%s\n", strerror(errno))) + return; + + memset(buf, 0, 9000); + buf[5000] = 0xaa; /* marker at offset 5000 (frag0) */ + + err = bpf_prog_test_run(prog_fd, 1, buf, 9000, + buf, &size, &retval, &duration); + + /* test_xdp_update_frags: buf[5000]: 0xaa -> 0xbb */ + CHECK(err || retval != XDP_PASS || buf[5000] != 0xbb, + "9000b", "err %d errno %d retval %d size %d\n", + err, errno, retval, size); + + free(buf); + + bpf_object__close(obj); +} + +void test_xdp_adjust_data(void) +{ + if (test__start_subtest("xdp_adjust_data")) + test_xdp_update_frag(); +} diff --git a/tools/testing/selftests/bpf/progs/test_xdp_update_frags.c b/tools/testing/selftests/bpf/progs/test_xdp_update_frags.c new file mode 100644 index 000000000000..2392cc3b6ba5 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/test_xdp_update_frags.c @@ -0,0 +1,49 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * This program is free software; you can redistribute it and/or + * modify it under the terms of version 2 of the GNU General Public + * License as published by the Free Software Foundation. + */ +#include +#include +#include + +int _version SEC("version") = 1; + +SEC("xdp_adjust_frags") +int _xdp_adjust_frags(struct xdp_md *xdp) +{ + __u8 *data_end = (void *)(long)xdp->data_end; + __u8 *data = (void *)(long)xdp->data; + __u32 offset = 5000; /* marker offset */ + int ret = XDP_DROP; + int data_len; + + if (data + sizeof(__u32) > data_end) + return XDP_DROP; + + data_len = bpf_xdp_adjust_data(xdp, offset); + if (data_len < 0) + return XDP_DROP; + + if (data_len > 5000) + goto out; + + data_end = (void *)(long)xdp->data_end; + data = (void *)(long)xdp->data; + offset -= data_len; /* offset in frag0 */ + + if (data + offset + 1 > data_end) + goto out; + + if (data[offset] != 0xaa) /* marker */ + goto out; + + data[offset] = 0xbb; /* update the marker */ + ret = XDP_PASS; +out: + bpf_xdp_adjust_data(xdp, 0); + return ret; +} + +char _license[] SEC("license") = "GPL";