From patchwork Tue Mar 17 17:29:22 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jesper Dangaard Brouer X-Patchwork-Id: 222381 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=3.0 tests=DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY, SPF_HELO_NONE, SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A7C03C2BB1D for ; Tue, 17 Mar 2020 17:29:44 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 6B69E20724 for ; Tue, 17 Mar 2020 17:29:44 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="RSkjJ5ur" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726781AbgCQR3n (ORCPT ); Tue, 17 Mar 2020 13:29:43 -0400 Received: from us-smtp-delivery-74.mimecast.com ([216.205.24.74]:41190 "EHLO us-smtp-delivery-74.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726754AbgCQR3l (ORCPT ); Tue, 17 Mar 2020 13:29:41 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1584466179; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=XT5j9w3tCbOpcwlhO1qxUNWifvfqKWYCIqKqorwEJME=; b=RSkjJ5urpzNIYFPUeyggwMkxHuphjjMizWTqzbPOVX2eBb/FDjGtsrTbXO6NFxUcRey3Tj cngQCDyQxVXcbbQa7RF+dEXwlV2yZ7MAVRW43luySi6S3h1O2h5dIcyEIy6WVhJ08CFIMw JA24O8kNqd+IxRVVoirdgs343jo9uIo= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-419-Ts0dC3DhPn6CkHhxZ4Hn_Q-1; Tue, 17 Mar 2020 13:29:35 -0400 X-MC-Unique: Ts0dC3DhPn6CkHhxZ4Hn_Q-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 3D723802C80; Tue, 17 Mar 2020 17:29:30 +0000 (UTC) Received: from firesoul.localdomain (unknown [10.40.208.26]) by smtp.corp.redhat.com (Postfix) with ESMTP id D27BC173FF; Tue, 17 Mar 2020 17:29:23 +0000 (UTC) Received: from [192.168.42.3] (localhost [IPv6:::1]) by firesoul.localdomain (Postfix) with ESMTP id EDF3F3074358B; Tue, 17 Mar 2020 18:29:22 +0100 (CET) Subject: [PATCH RFC v1 03/15] bnxt: add XDP frame size to driver From: Jesper Dangaard Brouer To: sameehj@amazon.com Cc: Michael Chan , Andy Gospodarek , Jesper Dangaard Brouer , netdev@vger.kernel.org, bpf@vger.kernel.org, zorik@amazon.com, akiyano@amazon.com, gtzalik@amazon.com, =?utf-8?b?VG9rZSBIw7hp?= =?utf-8?q?land-J=C3=B8rgensen?= , Daniel Borkmann , Alexei Starovoitov , John Fastabend , Alexander Duyck , Jeff Kirsher , David Ahern , Willem de Bruijn , Ilias Apalodimas , Lorenzo Bianconi Date: Tue, 17 Mar 2020 18:29:22 +0100 Message-ID: <158446616289.702578.7889111879119296431.stgit@firesoul> In-Reply-To: <158446612466.702578.2795159620575737080.stgit@firesoul> References: <158446612466.702578.2795159620575737080.stgit@firesoul> User-Agent: StGit/0.19 MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org This driver uses full PAGE_SIZE pages when XDP is enabled. Cc: Michael Chan Cc: Andy Gospodarek Signed-off-by: Jesper Dangaard Brouer --- drivers/net/ethernet/broadcom/bnxt/bnxt_xdp.c | 1 + 1 file changed, 1 insertion(+) diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt_xdp.c b/drivers/net/ethernet/broadcom/bnxt/bnxt_xdp.c index c6f6f2033880..5e3b4a3b69ea 100644 --- a/drivers/net/ethernet/broadcom/bnxt/bnxt_xdp.c +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt_xdp.c @@ -138,6 +138,7 @@ bool bnxt_rx_xdp(struct bnxt *bp, struct bnxt_rx_ring_info *rxr, u16 cons, xdp_set_data_meta_invalid(&xdp); xdp.data_end = *data_ptr + *len; xdp.rxq = &rxr->xdp_rxq; + xdp.frame_sz = PAGE_SIZE; /* BNXT_RX_PAGE_MODE(bp) when XDP enabled */ orig_data = xdp.data; rcu_read_lock(); From patchwork Tue Mar 17 17:29:28 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jesper Dangaard Brouer X-Patchwork-Id: 222382 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=3.0 tests=DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY, SPF_HELO_NONE, SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 819B5C2BB1D for ; Tue, 17 Mar 2020 17:29:41 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 5BFA920724 for ; Tue, 17 Mar 2020 17:29:41 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="YMU5e0J6" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726763AbgCQR3k (ORCPT ); Tue, 17 Mar 2020 13:29:40 -0400 Received: from us-smtp-delivery-74.mimecast.com ([216.205.24.74]:55490 "EHLO us-smtp-delivery-74.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726157AbgCQR3k (ORCPT ); Tue, 17 Mar 2020 13:29:40 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1584466178; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=3AsaCSGIzYZvw4mOnWKfUdq3Y1S7ifGzwLeXDOIt/dU=; b=YMU5e0J6xDyiXK6lJH5xy8ISc+4W0gOnlu8MxgRceQsHVrzsx9GUcvB4GTJBAee4UXkFUt 5QhXjXkUYTr1EFD9Z3DXXL9r/GH5X6J52Q3173VXe+Q7uCSjETAooe6pfE1P5+1KqO498v ASsd3I5AhGhtoWNJrUpybXOYY3RgisY= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-329-EWu_ZqJ-N-u2kNx6tjLbvQ-1; Tue, 17 Mar 2020 13:29:34 -0400 X-MC-Unique: EWu_ZqJ-N-u2kNx6tjLbvQ-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.phx2.redhat.com [10.5.11.22]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 03F9380259D; Tue, 17 Mar 2020 17:29:30 +0000 (UTC) Received: from firesoul.localdomain (unknown [10.40.208.26]) by smtp.corp.redhat.com (Postfix) with ESMTP id 1029610027B5; Tue, 17 Mar 2020 17:29:29 +0000 (UTC) Received: from [192.168.42.3] (localhost [IPv6:::1]) by firesoul.localdomain (Postfix) with ESMTP id 1075930721A66; Tue, 17 Mar 2020 18:29:28 +0100 (CET) Subject: [PATCH RFC v1 04/15] ixgbe: fix XDP redirect on archs with PAGE_SIZE above 4K From: Jesper Dangaard Brouer To: sameehj@amazon.com Cc: Jeff Kirsher , Jesper Dangaard Brouer , netdev@vger.kernel.org, bpf@vger.kernel.org, zorik@amazon.com, akiyano@amazon.com, gtzalik@amazon.com, =?utf-8?b?VG9rZSBIw7hp?= =?utf-8?q?land-J=C3=B8rgensen?= , Daniel Borkmann , Alexei Starovoitov , John Fastabend , Alexander Duyck , Jeff Kirsher , David Ahern , Willem de Bruijn , Ilias Apalodimas , Lorenzo Bianconi Date: Tue, 17 Mar 2020 18:29:28 +0100 Message-ID: <158446616798.702578.13193322070882482316.stgit@firesoul> In-Reply-To: <158446612466.702578.2795159620575737080.stgit@firesoul> References: <158446612466.702578.2795159620575737080.stgit@firesoul> User-Agent: StGit/0.19 MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.22 Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org The ixgbe driver have another memory model when compiled on archs with PAGE_SIZE above 4096 bytes. In this mode it doesn't split the page in two halves, but instead increment rx_buffer->page_offset by truesize of packet (which include headroom and tailroom for skb_shared_info). This is done correctly in ixgbe_build_skb(), but in ixgbe_rx_buffer_flip which is currently only called on XDP_TX and XDP_REDIRECT, it forgets to add the tailroom for skb_shared_info. This breaks XDP_REDIRECT, for veth and cpumap. Fix by adding size of skb_shared_info tailroom. Maintainers notice: This fix have been queued to Jeff. Fixes: 6453073987ba ("ixgbe: add initial support for xdp redirect") Cc: Jeff Kirsher Signed-off-by: Jesper Dangaard Brouer --- drivers/net/ethernet/intel/ixgbe/ixgbe_main.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c b/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c index 718931d951bc..ea6834bae04c 100644 --- a/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c +++ b/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c @@ -2254,7 +2254,8 @@ static void ixgbe_rx_buffer_flip(struct ixgbe_ring *rx_ring, rx_buffer->page_offset ^= truesize; #else unsigned int truesize = ring_uses_build_skb(rx_ring) ? - SKB_DATA_ALIGN(IXGBE_SKB_PAD + size) : + SKB_DATA_ALIGN(IXGBE_SKB_PAD + size) + + SKB_DATA_ALIGN(sizeof(struct skb_shared_info)) : SKB_DATA_ALIGN(size); rx_buffer->page_offset += truesize; From patchwork Tue Mar 17 17:29:38 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jesper Dangaard Brouer X-Patchwork-Id: 222380 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=3.0 tests=DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY, SPF_HELO_NONE, SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 411F1C5ACD6 for ; Tue, 17 Mar 2020 17:29:59 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 0D1ED20738 for ; Tue, 17 Mar 2020 17:29:59 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="BT4fanIL" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726810AbgCQR36 (ORCPT ); Tue, 17 Mar 2020 13:29:58 -0400 Received: from us-smtp-delivery-74.mimecast.com ([216.205.24.74]:26294 "EHLO us-smtp-delivery-74.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726793AbgCQR35 (ORCPT ); Tue, 17 Mar 2020 13:29:57 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1584466196; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=i8ADgiXCNrjUwtxx6VSppVeHIxTSGzYqm7aWz4SrIA0=; b=BT4fanIL02AWmoziPxMAID/TStz+lSvSxyEaUdwvs5n0wMyHjDacmFTX6MJ/4Ot6MWEhLj lVo2xxaMZfXjPYGKgu5eMgYcrrNo9zl/ug2A28S14HxHprYkQpyI5nKb8o4aAs0+oceV6o LreeCqWRE/ra019KK5YVJMakHr4jpYc= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-23-mgRM24zENyqrL7KKCXJgIw-1; Tue, 17 Mar 2020 13:29:52 -0400 X-MC-Unique: mgRM24zENyqrL7KKCXJgIw-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id C349CDB23; Tue, 17 Mar 2020 17:29:39 +0000 (UTC) Received: from firesoul.localdomain (unknown [10.40.208.26]) by smtp.corp.redhat.com (Postfix) with ESMTP id 3657260BEE; Tue, 17 Mar 2020 17:29:39 +0000 (UTC) Received: from [192.168.42.3] (localhost [IPv6:::1]) by firesoul.localdomain (Postfix) with ESMTP id 3B73D30721A66; Tue, 17 Mar 2020 18:29:38 +0100 (CET) Subject: [PATCH RFC v1 06/15] sfc: fix XDP-redirect in this driver From: Jesper Dangaard Brouer To: sameehj@amazon.com Cc: Jesper Dangaard Brouer , netdev@vger.kernel.org, bpf@vger.kernel.org, zorik@amazon.com, akiyano@amazon.com, gtzalik@amazon.com, =?utf-8?q?Toke_H=C3=B8iland-J=C3=B8rgensen?= , Daniel Borkmann , Alexei Starovoitov , John Fastabend , Alexander Duyck , Jeff Kirsher , David Ahern , Willem de Bruijn , Ilias Apalodimas , Lorenzo Bianconi Date: Tue, 17 Mar 2020 18:29:38 +0100 Message-ID: <158446617817.702578.15401678499421072150.stgit@firesoul> In-Reply-To: <158446612466.702578.2795159620575737080.stgit@firesoul> References: <158446612466.702578.2795159620575737080.stgit@firesoul> User-Agent: StGit/0.19 MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org TODO: Drop as already accepted in net-next as commit 86e85bf6981c ("sfc: fix XDP-redirect in this driver") XDP-redirect is broken in this driver sfc. XDP_REDIRECT requires tailroom for skb_shared_info when creating an SKB based on the redirected xdp_frame (both in cpumap and veth). The fix requires some initial explaining. The driver uses RX page-split when possible. It reserves the top 64 bytes in the RX-page for storing dma_addr (struct efx_rx_page_state). It also have the XDP recommended headroom of XDP_PACKET_HEADROOM (256 bytes). As it doesn't reserve any tailroom, it can still fit two standard MTU (1500) frames into one page. The sizeof struct skb_shared_info in 320 bytes. Thus drivers like ixgbe and i40e, reduce their XDP headroom to 192 bytes, which allows them to fit two frames with max 1536 bytes into a 4K page (192+1536+320=2048). The fix is to reduce this drivers headroom to 128 bytes and add the 320 bytes tailroom. This account for reserved top 64 bytes in the page, and still fit two frame in a page for normal MTUs. We must never go below 128 bytes of headroom for XDP, as one cacheline is for xdp_frame area and next cacheline is reserved for metadata area. Fixes: eb9a36be7f3e ("sfc: perform XDP processing on received packets") Signed-off-by: Jesper Dangaard Brouer --- drivers/net/ethernet/sfc/efx_common.c | 9 +++++---- drivers/net/ethernet/sfc/net_driver.h | 6 ++++++ drivers/net/ethernet/sfc/rx.c | 2 +- drivers/net/ethernet/sfc/rx_common.c | 6 +++--- 4 files changed, 15 insertions(+), 8 deletions(-) diff --git a/drivers/net/ethernet/sfc/efx_common.c b/drivers/net/ethernet/sfc/efx_common.c index b0d76bc19673..1799ff9a45d9 100644 --- a/drivers/net/ethernet/sfc/efx_common.c +++ b/drivers/net/ethernet/sfc/efx_common.c @@ -200,11 +200,11 @@ void efx_link_status_changed(struct efx_nic *efx) unsigned int efx_xdp_max_mtu(struct efx_nic *efx) { /* The maximum MTU that we can fit in a single page, allowing for - * framing, overhead and XDP headroom. + * framing, overhead and XDP headroom + tailroom. */ int overhead = EFX_MAX_FRAME_LEN(0) + sizeof(struct efx_rx_page_state) + efx->rx_prefix_size + efx->type->rx_buffer_padding + - efx->rx_ip_align + XDP_PACKET_HEADROOM; + efx->rx_ip_align + EFX_XDP_HEADROOM + EFX_XDP_TAILROOM; return PAGE_SIZE - overhead; } @@ -302,8 +302,9 @@ static void efx_start_datapath(struct efx_nic *efx) efx->rx_dma_len = (efx->rx_prefix_size + EFX_MAX_FRAME_LEN(efx->net_dev->mtu) + efx->type->rx_buffer_padding); - rx_buf_len = (sizeof(struct efx_rx_page_state) + XDP_PACKET_HEADROOM + - efx->rx_ip_align + efx->rx_dma_len); + rx_buf_len = (sizeof(struct efx_rx_page_state) + EFX_XDP_HEADROOM + + efx->rx_ip_align + efx->rx_dma_len + EFX_XDP_TAILROOM); + if (rx_buf_len <= PAGE_SIZE) { efx->rx_scatter = efx->type->always_rx_scatter; efx->rx_buffer_order = 0; diff --git a/drivers/net/ethernet/sfc/net_driver.h b/drivers/net/ethernet/sfc/net_driver.h index 9f9886f222c8..f96b1f9fe119 100644 --- a/drivers/net/ethernet/sfc/net_driver.h +++ b/drivers/net/ethernet/sfc/net_driver.h @@ -91,6 +91,12 @@ #define EFX_RX_BUF_ALIGNMENT 4 #endif +/* Non-standard XDP_PACKET_HEADROOM and tailroom to satisfy XDP_REDIRECT and + * still fit two standard MTU size packets into a single 4K page. + */ +#define EFX_XDP_HEADROOM 128 +#define EFX_XDP_TAILROOM SKB_DATA_ALIGN(sizeof(struct skb_shared_info)) + /* Forward declare Precision Time Protocol (PTP) support structure. */ struct efx_ptp_data; struct hwtstamp_config; diff --git a/drivers/net/ethernet/sfc/rx.c b/drivers/net/ethernet/sfc/rx.c index a2042f16babc..260352d97d9d 100644 --- a/drivers/net/ethernet/sfc/rx.c +++ b/drivers/net/ethernet/sfc/rx.c @@ -302,7 +302,7 @@ static bool efx_do_xdp(struct efx_nic *efx, struct efx_channel *channel, efx->rx_prefix_size); xdp.data = *ehp; - xdp.data_hard_start = xdp.data - XDP_PACKET_HEADROOM; + xdp.data_hard_start = xdp.data - EFX_XDP_HEADROOM; /* No support yet for XDP metadata */ xdp_set_data_meta_invalid(&xdp); diff --git a/drivers/net/ethernet/sfc/rx_common.c b/drivers/net/ethernet/sfc/rx_common.c index ee8beb87bdc1..e10c23833515 100644 --- a/drivers/net/ethernet/sfc/rx_common.c +++ b/drivers/net/ethernet/sfc/rx_common.c @@ -412,10 +412,10 @@ static int efx_init_rx_buffers(struct efx_rx_queue *rx_queue, bool atomic) index = rx_queue->added_count & rx_queue->ptr_mask; rx_buf = efx_rx_buffer(rx_queue, index); rx_buf->dma_addr = dma_addr + efx->rx_ip_align + - XDP_PACKET_HEADROOM; + EFX_XDP_HEADROOM; rx_buf->page = page; rx_buf->page_offset = page_offset + efx->rx_ip_align + - XDP_PACKET_HEADROOM; + EFX_XDP_HEADROOM; rx_buf->len = efx->rx_dma_len; rx_buf->flags = 0; ++rx_queue->added_count; @@ -433,7 +433,7 @@ static int efx_init_rx_buffers(struct efx_rx_queue *rx_queue, bool atomic) void efx_rx_config_page_split(struct efx_nic *efx) { efx->rx_page_buf_step = ALIGN(efx->rx_dma_len + efx->rx_ip_align + - XDP_PACKET_HEADROOM, + EFX_XDP_HEADROOM + EFX_XDP_TAILROOM, EFX_RX_BUF_ALIGNMENT); efx->rx_bufs_per_page = efx->rx_buffer_order ? 1 : ((PAGE_SIZE - sizeof(struct efx_rx_page_state)) / From patchwork Tue Mar 17 17:29:48 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jesper Dangaard Brouer X-Patchwork-Id: 222379 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=3.0 tests=DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY, SPF_HELO_NONE, SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5332FC1975A for ; Tue, 17 Mar 2020 17:30:10 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 2B48720724 for ; Tue, 17 Mar 2020 17:30:10 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="SxT2Quli" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726838AbgCQRaJ (ORCPT ); Tue, 17 Mar 2020 13:30:09 -0400 Received: from us-smtp-delivery-74.mimecast.com ([63.128.21.74]:53799 "EHLO us-smtp-delivery-74.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726386AbgCQRaJ (ORCPT ); Tue, 17 Mar 2020 13:30:09 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1584466208; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=xV8+EDBHRqVwSckpJwc2UUAHmVFdfl6/DmYv+cI16LU=; b=SxT2Quli8WXtzJLaiSDAkTUzYuWpQg2dX6S9cD0pXWQ8agFN3fxdlj1aDXkie5fNhAGvdA +gMWFAOCTz/B2ovGkCabhRp0Xom2MVepJBc6CcOZssdt/UKPI+wEuFXjLfPEB7zMauNK73 0kabbhpwJZgVmyt80Olhtkn7+0VkWug= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-132-aJbI5kJGPAyYKPulzcYGpA-1; Tue, 17 Mar 2020 13:30:04 -0400 X-MC-Unique: aJbI5kJGPAyYKPulzcYGpA-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.phx2.redhat.com [10.5.11.22]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id D3365A899F6; Tue, 17 Mar 2020 17:29:49 +0000 (UTC) Received: from firesoul.localdomain (unknown [10.40.208.26]) by smtp.corp.redhat.com (Postfix) with ESMTP id 5A2721001DC0; Tue, 17 Mar 2020 17:29:49 +0000 (UTC) Received: from [192.168.42.3] (localhost [IPv6:::1]) by firesoul.localdomain (Postfix) with ESMTP id 6521730721A66; Tue, 17 Mar 2020 18:29:48 +0100 (CET) Subject: [PATCH RFC v1 08/15] xdp: allow bpf_xdp_adjust_tail() to grow packet size From: Jesper Dangaard Brouer To: sameehj@amazon.com Cc: Jesper Dangaard Brouer , netdev@vger.kernel.org, bpf@vger.kernel.org, zorik@amazon.com, akiyano@amazon.com, gtzalik@amazon.com, =?utf-8?q?Toke_H=C3=B8iland-J=C3=B8rgensen?= , Daniel Borkmann , Alexei Starovoitov , John Fastabend , Alexander Duyck , Jeff Kirsher , David Ahern , Willem de Bruijn , Ilias Apalodimas , Lorenzo Bianconi Date: Tue, 17 Mar 2020 18:29:48 +0100 Message-ID: <158446618833.702578.690455723873770827.stgit@firesoul> In-Reply-To: <158446612466.702578.2795159620575737080.stgit@firesoul> References: <158446612466.702578.2795159620575737080.stgit@firesoul> User-Agent: StGit/0.19 MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.22 Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org FIXME: This patch MUST be LAST after all drivers have been updated!!! Finally, after all drivers have a frame size, allow BPF-helper bpf_xdp_adjust_tail() to grow or extend packet size at frame tail. Remember that helper/macro xdp_data_hard_end have reserved some tailroom. Thus, this helper makes sure that the BPF-prog don't have access to this tailroom area. Signed-off-by: Jesper Dangaard Brouer --- include/uapi/linux/bpf.h | 4 ++-- net/core/filter.c | 18 ++++++++++++++++-- 2 files changed, 18 insertions(+), 4 deletions(-) diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h index 14dc4f9fb3c8..1fa6ab616ec3 100644 --- a/include/uapi/linux/bpf.h +++ b/include/uapi/linux/bpf.h @@ -1942,8 +1942,8 @@ union bpf_attr { * int bpf_xdp_adjust_tail(struct xdp_buff *xdp_md, int delta) * Description * Adjust (move) *xdp_md*\ **->data_end** by *delta* bytes. It is - * only possible to shrink the packet as of this writing, - * therefore *delta* must be a negative integer. + * possible to both shrink and grow the packet tail. + * Shrink done via *delta* being a negative integer. * * A call to this helper is susceptible to change the underlying * packet buffer. Therefore, at load time, all checks on pointers diff --git a/net/core/filter.c b/net/core/filter.c index 4a08c9fb2be7..0ceddee0c678 100644 --- a/net/core/filter.c +++ b/net/core/filter.c @@ -3409,12 +3409,26 @@ static const struct bpf_func_proto bpf_xdp_adjust_head_proto = { BPF_CALL_2(bpf_xdp_adjust_tail, struct xdp_buff *, xdp, int, offset) { + void *data_hard_end = xdp_data_hard_end(xdp); void *data_end = xdp->data_end + offset; - /* only shrinking is allowed for now. */ - if (unlikely(offset >= 0)) + /* Notice that xdp_data_hard_end have reserved some tailroom */ + if (unlikely(data_end >= data_hard_end)) return -EINVAL; + /* DANGER: ALL drivers MUST be converted to init xdp->frame_sz + * - Adding some chicken checks below + * - Will (likely) not be for upstream + */ + if (unlikely(xdp->frame_sz < (xdp->data_end - xdp->data_hard_start))) { + WARN(1, "Too small xdp->frame_sz = %d\n", xdp->frame_sz); + return -EINVAL; + } + if (unlikely(xdp->frame_sz > PAGE_SIZE)) { + WARN(1, "Too BIG xdp->frame_sz = %d\n", xdp->frame_sz); + return -EINVAL; + } + if (unlikely(data_end < xdp->data + ETH_HLEN)) return -EINVAL; From patchwork Tue Mar 17 17:30:03 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jesper Dangaard Brouer X-Patchwork-Id: 222378 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=3.0 tests=DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY, SPF_HELO_NONE, SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9C973C5ACD6 for ; Tue, 17 Mar 2020 17:30:22 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 74C7B20724 for ; Tue, 17 Mar 2020 17:30:22 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="cpNGEMYp" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726851AbgCQRaV (ORCPT ); Tue, 17 Mar 2020 13:30:21 -0400 Received: from us-smtp-delivery-74.mimecast.com ([216.205.24.74]:53390 "EHLO us-smtp-delivery-74.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726647AbgCQRaV (ORCPT ); Tue, 17 Mar 2020 13:30:21 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1584466219; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=NDjKx6IwnF/NmrEYCxfuH2dwTT6/S8zsEJsWIeaXprg=; b=cpNGEMYpFryqIs7iDVOGhP4KrJvGRgynyB6weIcivNEYeRPOjzqG9f+YHNWoGVmK2YaqUw oQq4fAlhRPOzsbrLdyN8vpr5ju27O5CaKhXEcf08xh0mRkAXK7lfckZSP7sWwcSvN52YbO Zi1eK/KrlbA3z+S54+ge4QqrQauWPaM= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-177-oNsI659-Nsu79eluQ9RN-A-1; Tue, 17 Mar 2020 13:30:15 -0400 X-MC-Unique: oNsI659-Nsu79eluQ9RN-A-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 25AAF8018B6; Tue, 17 Mar 2020 17:30:05 +0000 (UTC) Received: from firesoul.localdomain (unknown [10.40.208.26]) by smtp.corp.redhat.com (Postfix) with ESMTP id ACB6760BEE; Tue, 17 Mar 2020 17:30:04 +0000 (UTC) Received: from [192.168.42.3] (localhost [IPv6:::1]) by firesoul.localdomain (Postfix) with ESMTP id A7E0030721A66; Tue, 17 Mar 2020 18:30:03 +0100 (CET) Subject: [PATCH RFC v1 11/15] xdp: xdp_frame add member frame_sz and handle in convert_to_xdp_frame From: Jesper Dangaard Brouer To: sameehj@amazon.com Cc: Jesper Dangaard Brouer , netdev@vger.kernel.org, bpf@vger.kernel.org, zorik@amazon.com, akiyano@amazon.com, gtzalik@amazon.com, =?utf-8?q?Toke_H=C3=B8iland-J=C3=B8rgensen?= , Daniel Borkmann , Alexei Starovoitov , John Fastabend , Alexander Duyck , Jeff Kirsher , David Ahern , Willem de Bruijn , Ilias Apalodimas , Lorenzo Bianconi Date: Tue, 17 Mar 2020 18:30:03 +0100 Message-ID: <158446620361.702578.11853795393121537634.stgit@firesoul> In-Reply-To: <158446612466.702578.2795159620575737080.stgit@firesoul> References: <158446612466.702578.2795159620575737080.stgit@firesoul> User-Agent: StGit/0.19 MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org Use hole in struct xdp_frame, when adding member frame_sz, which keeps same sizeof struct (32 bytes) Drivers ixgbe and sfc had bug cases where the necessary/expected tailroom was not reserved. This can lead to some hard to catch memory corruption issues. Having the drivers frame_sz this can be detected when packet length/end via xdp->data_end exceed the xdp_data_hard_end pointer, which accounts for the reserved the tailroom. When detecting this driver issue, simply fail the conversion with NULL, which results in feedback to driver (failing xdp_do_redirect()) causing driver to drop packet. Given the lack of consistent XDP stats, this can be hard to troubleshoot. And given this is a driver bug, we want to generate some more noise in form of a WARN stack dump (to ID the driver code that inlined convert_to_xdp_frame). Inlining the WARN macro is problematic, because it adds an asm instruction (on Intel CPUs ud2) what influence instruction cache prefetching. Thus, introduce xdp_warn and macro XDP_WARN, to avoid this and at the same time make identifying the function and line of this inlined function easier. Signed-off-by: Jesper Dangaard Brouer --- include/net/xdp.h | 14 +++++++++++++- net/core/xdp.c | 7 +++++++ 2 files changed, 20 insertions(+), 1 deletion(-) diff --git a/include/net/xdp.h b/include/net/xdp.h index 99f4374f6214..55a885aa4e53 100644 --- a/include/net/xdp.h +++ b/include/net/xdp.h @@ -93,7 +93,8 @@ struct xdp_frame { void *data; u16 len; u16 headroom; - u16 metasize; + u32 metasize:8; + u32 frame_sz:24; /* Lifetime of xdp_rxq_info is limited to NAPI/enqueue time, * while mem info is valid on remote CPU. */ @@ -108,6 +109,10 @@ static inline void xdp_scrub_frame(struct xdp_frame *frame) frame->dev_rx = NULL; } +/* Avoids inlining WARN macro in fast-path */ +void xdp_warn(const char* msg, const char* func, const int line); +#define XDP_WARN(msg) xdp_warn(msg, __func__, __LINE__) + struct xdp_frame *xdp_convert_zc_to_xdp_frame(struct xdp_buff *xdp); /* Convert xdp_buff to xdp_frame */ @@ -128,6 +133,12 @@ struct xdp_frame *convert_to_xdp_frame(struct xdp_buff *xdp) if (unlikely((headroom - metasize) < sizeof(*xdp_frame))) return NULL; + /* Catch if driver didn't reserve tailroom for skb_shared_info */ + if (unlikely(xdp->data_end > xdp_data_hard_end(xdp))) { + XDP_WARN("Driver BUG: missing reserved tailroom"); + return NULL; + } + /* Store info in top of packet */ xdp_frame = xdp->data_hard_start; @@ -135,6 +146,7 @@ struct xdp_frame *convert_to_xdp_frame(struct xdp_buff *xdp) xdp_frame->len = xdp->data_end - xdp->data; xdp_frame->headroom = headroom - sizeof(*xdp_frame); xdp_frame->metasize = metasize; + xdp_frame->frame_sz = xdp->frame_sz; /* rxq only valid until napi_schedule ends, convert to xdp_mem_info */ xdp_frame->mem = xdp->rxq->mem; diff --git a/net/core/xdp.c b/net/core/xdp.c index 4c7ea85486af..4bc3026ae218 100644 --- a/net/core/xdp.c +++ b/net/core/xdp.c @@ -11,6 +11,7 @@ #include #include #include +#include #include #include @@ -496,3 +497,9 @@ struct xdp_frame *xdp_convert_zc_to_xdp_frame(struct xdp_buff *xdp) return xdpf; } EXPORT_SYMBOL_GPL(xdp_convert_zc_to_xdp_frame); + +/* Used by XDP_WARN macro, to avoid inlining WARN() in fast-path */ +void xdp_warn(const char* msg, const char* func, const int line) { + WARN(1, "XDP_WARN: %s(line:%d): %s\n", func, line, msg); +}; +EXPORT_SYMBOL_GPL(xdp_warn); From patchwork Tue Mar 17 17:30:08 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jesper Dangaard Brouer X-Patchwork-Id: 222377 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=3.0 tests=DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY, SPF_HELO_NONE, SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 72368C1975A for ; Tue, 17 Mar 2020 17:30:32 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 4927720735 for ; Tue, 17 Mar 2020 17:30:32 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="aYLJNpJU" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726874AbgCQRab (ORCPT ); Tue, 17 Mar 2020 13:30:31 -0400 Received: from us-smtp-delivery-74.mimecast.com ([216.205.24.74]:37439 "EHLO us-smtp-delivery-74.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726747AbgCQRab (ORCPT ); Tue, 17 Mar 2020 13:30:31 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1584466230; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ZzzjCBq9WyI3Bb6b0nAFkEUdPjhrJ6l3Fs8nfiWFreM=; b=aYLJNpJU35f6vkSJdEHPvdQTfaNz6AOz+P7xMyX58t3TMeCoeXy5kqCa4qgTqUD4s3V9qY tQm4PnujM+s/A+Bcw2RWxQig48rgsfipPJ5RybNqaSAdbvtVujGUSOOqKCBLRMznNiR7wP Gg1iDOEfdOt7O0EVGKHLTOjzF+NeF5k= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-465-eeQK3tLeOlapXfPLDVNk1w-1; Tue, 17 Mar 2020 13:30:26 -0400 X-MC-Unique: eeQK3tLeOlapXfPLDVNk1w-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 7799F8018D9; Tue, 17 Mar 2020 17:30:10 +0000 (UTC) Received: from firesoul.localdomain (unknown [10.40.208.26]) by smtp.corp.redhat.com (Postfix) with ESMTP id B099960BEE; Tue, 17 Mar 2020 17:30:09 +0000 (UTC) Received: from [192.168.42.3] (localhost [IPv6:::1]) by firesoul.localdomain (Postfix) with ESMTP id BCD4B30721A66; Tue, 17 Mar 2020 18:30:08 +0100 (CET) Subject: [PATCH RFC v1 12/15] xdp: cpumap redirect use frame_sz and increase skb_tailroom From: Jesper Dangaard Brouer To: sameehj@amazon.com Cc: Jesper Dangaard Brouer , netdev@vger.kernel.org, bpf@vger.kernel.org, zorik@amazon.com, akiyano@amazon.com, gtzalik@amazon.com, =?utf-8?q?Toke_H=C3=B8iland-J=C3=B8rgensen?= , Daniel Borkmann , Alexei Starovoitov , John Fastabend , Alexander Duyck , Jeff Kirsher , David Ahern , Willem de Bruijn , Ilias Apalodimas , Lorenzo Bianconi Date: Tue, 17 Mar 2020 18:30:08 +0100 Message-ID: <158446620869.702578.7089227022464284959.stgit@firesoul> In-Reply-To: <158446612466.702578.2795159620575737080.stgit@firesoul> References: <158446612466.702578.2795159620575737080.stgit@firesoul> User-Agent: StGit/0.19 MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org Knowing the memory size backing the packet/xdp_frame data area, and knowing it already have reserved room for skb_shared_info, simplifies using build_skb significantly. With this change we no-longer lie about the SKB truesize, but more importantly a significant larger skb_tailroom is now provided, e.g. when drivers uses a full PAGE_SIZE. This extra tailroom (in linear area) can be used by the network stack when coalescing SKBs (e.g. in skb_try_coalesce, see TCP cases where tcp_queue_rcv() can 'eat' skb). Signed-off-by: Jesper Dangaard Brouer --- kernel/bpf/cpumap.c | 21 +++------------------ 1 file changed, 3 insertions(+), 18 deletions(-) diff --git a/kernel/bpf/cpumap.c b/kernel/bpf/cpumap.c index 70f71b154fa5..9c777ac4d4bd 100644 --- a/kernel/bpf/cpumap.c +++ b/kernel/bpf/cpumap.c @@ -162,25 +162,10 @@ static struct sk_buff *cpu_map_build_skb(struct bpf_cpu_map_entry *rcpu, /* Part of headroom was reserved to xdpf */ hard_start_headroom = sizeof(struct xdp_frame) + xdpf->headroom; - /* build_skb need to place skb_shared_info after SKB end, and - * also want to know the memory "truesize". Thus, need to - * know the memory frame size backing xdp_buff. - * - * XDP was designed to have PAGE_SIZE frames, but this - * assumption is not longer true with ixgbe and i40e. It - * would be preferred to set frame_size to 2048 or 4096 - * depending on the driver. - * frame_size = 2048; - * frame_len = frame_size - sizeof(*xdp_frame); - * - * Instead, with info avail, skb_shared_info in placed after - * packet len. This, unfortunately fakes the truesize. - * Another disadvantage of this approach, the skb_shared_info - * is not at a fixed memory location, with mixed length - * packets, which is bad for cache-line hotness. + /* Memory size backing xdp_frame data already have reserved + * room for build_skb to place skb_shared_info in tailroom. */ - frame_size = SKB_DATA_ALIGN(xdpf->len + hard_start_headroom) + - SKB_DATA_ALIGN(sizeof(struct skb_shared_info)); + frame_size = xdpf->frame_sz; pkt_data_start = xdpf->data - hard_start_headroom; skb = build_skb_around(skb, pkt_data_start, frame_size); From patchwork Tue Mar 17 17:30:18 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jesper Dangaard Brouer X-Patchwork-Id: 222375 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=3.0 tests=DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY, SPF_HELO_NONE, SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 699D2C5ACD6 for ; Tue, 17 Mar 2020 17:30:51 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 4479520724 for ; Tue, 17 Mar 2020 17:30:51 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="Pd4LwQ+n" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726979AbgCQRau (ORCPT ); Tue, 17 Mar 2020 13:30:50 -0400 Received: from us-smtp-delivery-74.mimecast.com ([216.205.24.74]:52854 "EHLO us-smtp-delivery-74.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726968AbgCQRas (ORCPT ); Tue, 17 Mar 2020 13:30:48 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1584466247; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Zp+AQxs94EVnMCSzOQE9xni13biyNXHyfWNC7YYNhrc=; b=Pd4LwQ+nJpeK+ZggLUNnXAWDtP6yBfImsW9bWfsP6xASyxmDALvogY5bcaspBUvVtG5znY XGHWwPDMD1y1hOgI+VDVxQxtyxW+K6GGzL2DhukQPUsm6YHsnXoFhWe9G8b524G0cP/26y NyeEVPJvEvviRo558PE7rRpPpw7Qu1k= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-89-6NdTwgLpOoiXgJ9UM3wdQA-1; Tue, 17 Mar 2020 13:30:41 -0400 X-MC-Unique: 6NdTwgLpOoiXgJ9UM3wdQA-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 49799801E67; Tue, 17 Mar 2020 17:30:20 +0000 (UTC) Received: from firesoul.localdomain (unknown [10.40.208.26]) by smtp.corp.redhat.com (Postfix) with ESMTP id DE7FC5C1BB; Tue, 17 Mar 2020 17:30:19 +0000 (UTC) Received: from [192.168.42.3] (localhost [IPv6:::1]) by firesoul.localdomain (Postfix) with ESMTP id E614630721A66; Tue, 17 Mar 2020 18:30:18 +0100 (CET) Subject: [PATCH RFC v1 14/15] veth: xdp using frame_sz in veth driver From: Jesper Dangaard Brouer To: sameehj@amazon.com Cc: Jesper Dangaard Brouer , netdev@vger.kernel.org, bpf@vger.kernel.org, zorik@amazon.com, akiyano@amazon.com, gtzalik@amazon.com, =?utf-8?q?Toke_H=C3=B8iland-J=C3=B8rgensen?= , Daniel Borkmann , Alexei Starovoitov , John Fastabend , Alexander Duyck , Jeff Kirsher , David Ahern , Willem de Bruijn , Ilias Apalodimas , Lorenzo Bianconi Date: Tue, 17 Mar 2020 18:30:18 +0100 Message-ID: <158446621887.702578.17234304084556809684.stgit@firesoul> In-Reply-To: <158446612466.702578.2795159620575737080.stgit@firesoul> References: <158446612466.702578.2795159620575737080.stgit@firesoul> User-Agent: StGit/0.19 MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org Signed-off-by: Jesper Dangaard Brouer --- drivers/net/veth.c | 15 +++++++++++++-- 1 file changed, 13 insertions(+), 2 deletions(-) diff --git a/drivers/net/veth.c b/drivers/net/veth.c index 8cdc4415fa70..be88625162cd 100644 --- a/drivers/net/veth.c +++ b/drivers/net/veth.c @@ -492,6 +492,8 @@ static struct sk_buff *veth_xdp_rcv_one(struct veth_rq *rq, unsigned int *xdp_xmit, struct veth_xdp_tx_bq *bq) { +// hmm... do we really need to exclude sizeof(struct xdp_frame) ? +// ... as bpf_xdp_adjust_head() already handle this. void *hard_start = frame->data - frame->headroom; void *head = hard_start - sizeof(struct xdp_frame); int len = frame->len, delta = 0; @@ -506,11 +508,12 @@ static struct sk_buff *veth_xdp_rcv_one(struct veth_rq *rq, struct xdp_buff xdp; u32 act; - xdp.data_hard_start = hard_start; + xdp.data_hard_start = hard_start; /* exclude xdp_frame area */ xdp.data = frame->data; xdp.data_end = frame->data + frame->len; xdp.data_meta = frame->data - frame->metasize; xdp.rxq = &rq->xdp_rxq; + xdp.frame_sz = frame->frame_sz - sizeof(struct xdp_frame); act = bpf_prog_run_xdp(xdp_prog, &xdp); @@ -639,6 +642,11 @@ static struct sk_buff *veth_xdp_rcv_skb(struct veth_rq *rq, struct sk_buff *skb, xdp.data_end = xdp.data + pktlen; xdp.data_meta = xdp.data; xdp.rxq = &rq->xdp_rxq; + + /* SKB "head" area always have tailroom for skb_shared_info */ + xdp.frame_sz = (void *)skb_end_pointer(skb) - xdp.data_hard_start; + xdp.frame_sz += SKB_DATA_ALIGN(sizeof(struct skb_shared_info)); + orig_data = xdp.data; orig_data_end = xdp.data_end; @@ -678,6 +686,7 @@ static struct sk_buff *veth_xdp_rcv_skb(struct veth_rq *rq, struct sk_buff *skb, } rcu_read_unlock(); + /* check if bpf_xdp_adjust_head was used */ delta = orig_data - xdp.data; off = mac_len + delta; if (off > 0) @@ -685,9 +694,11 @@ static struct sk_buff *veth_xdp_rcv_skb(struct veth_rq *rq, struct sk_buff *skb, else if (off < 0) __skb_pull(skb, -off); skb->mac_header -= delta; + + /* check if bpf_xdp_adjust_tail was used */ off = xdp.data_end - orig_data_end; if (off != 0) - __skb_put(skb, off); + __skb_put(skb, off); /* positive on grow, negative on shrink */ skb->protocol = eth_type_trans(skb, rq->dev); metalen = xdp.data - xdp.data_meta; From patchwork Tue Mar 17 17:30:24 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jesper Dangaard Brouer X-Patchwork-Id: 222376 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=3.0 tests=DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY, SPF_HELO_NONE, SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6D070C5ACD6 for ; Tue, 17 Mar 2020 17:30:48 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 47FD020724 for ; Tue, 17 Mar 2020 17:30:48 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="LYyA4C0B" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726965AbgCQRar (ORCPT ); Tue, 17 Mar 2020 13:30:47 -0400 Received: from us-smtp-delivery-74.mimecast.com ([63.128.21.74]:50688 "EHLO us-smtp-delivery-74.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726738AbgCQRaq (ORCPT ); Tue, 17 Mar 2020 13:30:46 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1584466245; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=hRA3mLxIRW8RlSqNs1JzRim2qfbZVa1Otm7sdJNcWjU=; b=LYyA4C0BJp7dJo6Ov00kNMHL//39t/Nw+dYB2svPv6Vr91dGJyPah56gygeQhTDo1zyzqp 9gFPkxG8GA17r+sNLIAuC4a9Y4VuwRUibJ6oWaA0DXuWBL5PW7Gsx5GWPz3m/AwpXTyREK JQpOUKaro8QaZcYTOaBcLjkl48Dwr5w= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-180-nkwSaraWPOS9d6OYjGG8vg-1; Tue, 17 Mar 2020 13:30:43 -0400 X-MC-Unique: nkwSaraWPOS9d6OYjGG8vg-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 6480B1085988; Tue, 17 Mar 2020 17:30:25 +0000 (UTC) Received: from firesoul.localdomain (unknown [10.40.208.26]) by smtp.corp.redhat.com (Postfix) with ESMTP id 06CD960BEE; Tue, 17 Mar 2020 17:30:25 +0000 (UTC) Received: from [192.168.42.3] (localhost [IPv6:::1]) by firesoul.localdomain (Postfix) with ESMTP id 0722930721A66; Tue, 17 Mar 2020 18:30:24 +0100 (CET) Subject: [PATCH RFC v1 15/15] dpaa2-eth: add XDP frame size From: Jesper Dangaard Brouer To: sameehj@amazon.com Cc: Jesper Dangaard Brouer , netdev@vger.kernel.org, bpf@vger.kernel.org, zorik@amazon.com, akiyano@amazon.com, gtzalik@amazon.com, =?utf-8?q?Toke_H=C3=B8iland-J=C3=B8rgensen?= , Daniel Borkmann , Alexei Starovoitov , John Fastabend , Alexander Duyck , Jeff Kirsher , David Ahern , Willem de Bruijn , Ilias Apalodimas , Lorenzo Bianconi Date: Tue, 17 Mar 2020 18:30:24 +0100 Message-ID: <158446622395.702578.14405139445906785865.stgit@firesoul> In-Reply-To: <158446612466.702578.2795159620575737080.stgit@firesoul> References: <158446612466.702578.2795159620575737080.stgit@firesoul> User-Agent: StGit/0.19 MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org This is the full page size: #define DPAA2_ETH_RX_BUF_RAW_SIZE PAGE_SIZE Signed-off-by: Jesper Dangaard Brouer --- drivers/net/ethernet/freescale/dpaa2/dpaa2-eth.c | 1 + 1 file changed, 1 insertion(+) diff --git a/drivers/net/ethernet/freescale/dpaa2/dpaa2-eth.c b/drivers/net/ethernet/freescale/dpaa2/dpaa2-eth.c index 7ff147e89426..8b03b38b33c9 100644 --- a/drivers/net/ethernet/freescale/dpaa2/dpaa2-eth.c +++ b/drivers/net/ethernet/freescale/dpaa2/dpaa2-eth.c @@ -301,6 +301,7 @@ static u32 run_xdp(struct dpaa2_eth_priv *priv, xdp.data_hard_start = xdp.data - XDP_PACKET_HEADROOM; xdp_set_data_meta_invalid(&xdp); xdp.rxq = &ch->xdp_rxq; + xdp.frame_sz = DPAA2_ETH_RX_BUF_RAW_SIZE; xdp_act = bpf_prog_run_xdp(xdp_prog, &xdp);