From patchwork Thu May 8 00:48:22 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mina Almasry X-Patchwork-Id: 888936 Received: from mail-pl1-f202.google.com (mail-pl1-f202.google.com [209.85.214.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 83D0E15B135 for ; Thu, 8 May 2025 00:48:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746665319; cv=none; b=sO5iZObe1hAkcPgDOX4nTDUFuqeosnQxpZ1xWPQ7hjeui22zmRZDq3LnW/CaB+Q87+2/gE5uSEr/t1BZM0fxHmuvp95gXwJHO3QsQXrvzw1UmmJuj1fWMfILeDcfkX+RSYkhlp8/7++n/An266+Pqe64Djou0Srg9Ig80pZiq5s= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746665319; c=relaxed/simple; bh=dyqcNTk9OPsrtIpqTgjziEIAqTB6qpkn/n/NvMbCZEI=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=F9e3qXxquXH5XfAKmml87LuD1wUK7MTPElf6zljn+v6Ha9yf8GGhFXyevDcKBDjU0A+kym3h2DjNRcvy2pjU7TeSjjtMYo70Kb17K44UEp2/ToRo4PzHX2qlbtlDe2ltJaY4hY7JclRj0uqcGLHVnqvPXbLLZFKBLcQfCQjjdf8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--almasrymina.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=GhWrb+C8; arc=none smtp.client-ip=209.85.214.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--almasrymina.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="GhWrb+C8" Received: by mail-pl1-f202.google.com with SMTP id d9443c01a7336-22e815dd332so7226975ad.3 for ; Wed, 07 May 2025 17:48:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1746665316; x=1747270116; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=kYDyR9CQ41BaDC42YOSlqGxpvCJv7HXaEYFXBgq5RBU=; b=GhWrb+C87m01B8jmhmUiMpBL5ofGhLPaZpZT5FU0q6P2pCTTJdzxPX9hGJZrq9R7Nq FCKopLlT7I2jH2MxdZOLvLL1pVNmsr27Ut9BwlPnMWXWO+Qlh938Nqsf+oL41JXNcuDl 0iyawIQ7SksEp3O7yh922KNRHIZQy0Zx6o9tv/cb5gZNGSI/oK2X4vypag7pXaGbbtTL 76bxmsSeWCbpRropEXW8GJgYNKXB0eYCYK9tcG9tYrUVeDxQe4HY+hbfynhhSx3nuNu4 kmH7s0G+o5iyJEeCAecY/CnVvc0RgEFKwA8UcRuEgaCiluwUncrwDPvT1NVe77QIGDjs iQkA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1746665316; x=1747270116; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=kYDyR9CQ41BaDC42YOSlqGxpvCJv7HXaEYFXBgq5RBU=; b=SWhB06gsZg8TA78Bk5RjaActjfpv61okh3+hofRjsP4rDGE2yXO/IsKN2z4Lt0ZyTn ks0ao4yHW7d/nARu66tLjc9WiJtM/oxf2twe6fgFApb+yLwuPxtKswroqgLmckLKqEBd UsfPY1LLK8S7P5gBwdGtN1IFSK+dNpurXR3/m40/FCRKt2FsThifkdsyPtCr8qHzyqAO gRUW34V2BR7wLGlwubFYT2LRPNz+S0T9zByfTT0CUSR5sXK/ZhxUKKaNLKQj8vq8fsi7 pCrwFdiOyeL6mZh3dhgByaEVjn2KHSicBlkZimwzVueGKjeVxiv99YKWo1ypOOADTXT+ aUXA== X-Forwarded-Encrypted: i=1; AJvYcCVDg1iUwYUGLQ5tOpoXD3AX8tlx/usZCOpXO459BJ6hQRxkNfNuUtwqOVP3mQbmK7J8vMNbT2IcEjRBhc7u+Ok=@vger.kernel.org X-Gm-Message-State: AOJu0YwKOIw15JcxpVGe8V+xzVYJb9do10KuuGca1gZUEsuEPpMFjouz ilhFkZTQTdmuCpapornS9e1nty2fAzQA9BA7zqhkqN75UwIR++M1vjL7WbjEl3lk7ToTwiCox76 0yXukA45g2fWmOONPlRuk/g== X-Google-Smtp-Source: AGHT+IFacogxr0ODO6PRxraF8YX4ITl4t0rUQ989b/4Z3MefJOwdb8bNyzEquX5FLpjdtycoEOAfqSD9j7c/SD4+Ug== X-Received: from plblg15.prod.google.com ([2002:a17:902:fb8f:b0:22e:4574:b2bc]) (user=almasrymina job=prod-delivery.src-stubby-dispatcher) by 2002:a17:902:db0c:b0:22e:5e05:b89d with SMTP id d9443c01a7336-22e5ea7caeamr77706365ad.16.1746665315831; Wed, 07 May 2025 17:48:35 -0700 (PDT) Date: Thu, 8 May 2025 00:48:22 +0000 In-Reply-To: <20250508004830.4100853-1-almasrymina@google.com> Precedence: bulk X-Mailing-List: linux-kselftest@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250508004830.4100853-1-almasrymina@google.com> X-Mailer: git-send-email 2.49.0.987.g0cc8ee98dc-goog Message-ID: <20250508004830.4100853-3-almasrymina@google.com> Subject: [PATCH net-next v14 2/9] net: add get_netmem/put_netmem support From: Mina Almasry To: netdev@vger.kernel.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, io-uring@vger.kernel.org, virtualization@lists.linux.dev, kvm@vger.kernel.org, linux-kselftest@vger.kernel.org Cc: Mina Almasry , Donald Hunter , Jakub Kicinski , "David S. Miller" , Eric Dumazet , Paolo Abeni , Simon Horman , Jonathan Corbet , Andrew Lunn , Jeroen de Borst , Harshitha Ramamurthy , Kuniyuki Iwashima , Willem de Bruijn , Jens Axboe , Pavel Begunkov , David Ahern , Neal Cardwell , "Michael S. Tsirkin" , Jason Wang , Xuan Zhuo , " =?utf-8?q?Eugenio_P=C3=A9rez?= " , Stefan Hajnoczi , Stefano Garzarella , Shuah Khan , sdf@fomichev.me, dw@davidwei.uk, Jamal Hadi Salim , Victor Nogueira , Pedro Tammela , Samiullah Khawaja Currently net_iovs support only pp ref counts, and do not support a page ref equivalent. This is fine for the RX path as net_iovs are used exclusively with the pp and only pp refcounting is needed there. The TX path however does not use pp ref counts, thus, support for get_page/put_page equivalent is needed for netmem. Support get_netmem/put_netmem. Check the type of the netmem before passing it to page or net_iov specific code to obtain a page ref equivalent. For dmabuf net_iovs, we obtain a ref on the underlying binding. This ensures the entire binding doesn't disappear until all the net_iovs have been put_netmem'ed. We do not need to track the refcount of individual dmabuf net_iovs as we don't allocate/free them from a pool similar to what the buddy allocator does for pages. This code is written to be extensible by other net_iov implementers. get_netmem/put_netmem will check the type of the netmem and route it to the correct helper: pages -> [get|put]_page() dmabuf net_iovs -> net_devmem_[get|put]_net_iov() new net_iovs -> new helpers Signed-off-by: Mina Almasry Acked-by: Stanislav Fomichev --- v5: https://lore.kernel.org/netdev/20250227041209.2031104-2-almasrymina@google.com/ - Updated to check that the net_iov is devmem before calling net_devmem_put_net_iov(). - Jakub requested that callers of __skb_frag_ref()/skb_page_unref be inspected to make sure that they generate / anticipate skbs with the correct pp_recycle and unreadable setting: skb_page_unref ============== - skb_page_unref is unreachable from these callers due to unreadable checks returning early: gro_pull_from_frag0, skb_copy_ubufs, __pskb_pull_tail - callers that are reachable for unreadable skbs. These would only see rx unreadable skbs with pp_recycle set before this patchset and would drop a pp ref count. After this patchset they can see tx unreadable skbs with no pp attached and no pp_recycle set, and so now they will drop a net_iov ref via put_netmem: __pskb_trim, __pskb_trim_head, skb_release_data, skb_shift __skb_frag_ref ============== Before this patchset __skb_frag_ref would not do the right thing if it saw any unreadable skbs, either with pp_recycle set or not. Because it unconditionally tries to acquire a page ref, but with RX only support I can't reproduce calls to __skb_frag_ref even after enabling tc forwarding to TX. After this patchset __skb_frag_ref would obtain a page ref equivalent on dmabuf net_iovs, by obtaining a ref on the binding. Callers that are unreachable for unreadable skbs: - veth_xdp_get Callers that are reachable for unreadable skbs, and from code review they look specific to the TX path: - tcp_grow_skb, __skb_zcopy_downgrade_managed, __pskb_copy_fclone, pskb_expand_head, skb_zerocopy, skb_split, pksb_carve_inside_header, pskb_care_inside_nonlinear, tcp_clone_payload, skb_segment. Callers that are reachable for unreadable skbs, and from code review they look reachable in the RX path, although my testing never hit these paths. These are concerning. Maybe we should put this patch in net and cc stable? However, no drivers currently enable unreadable netmem, so fixing this in net-next is fine as well maybe: - skb_shift, skb_try_coalesce v2: - Add comment on top of refcount_t ref explaining the usage in the XT path. - Fix missing definition of net_devmem_dmabuf_binding_put in this patch. --- include/linux/skbuff_ref.h | 4 ++-- include/net/netmem.h | 3 +++ net/core/devmem.c | 10 ++++++++++ net/core/devmem.h | 20 ++++++++++++++++++++ net/core/skbuff.c | 30 ++++++++++++++++++++++++++++++ 5 files changed, 65 insertions(+), 2 deletions(-) diff --git a/include/linux/skbuff_ref.h b/include/linux/skbuff_ref.h index 0f3c58007488a..9e49372ef1a05 100644 --- a/include/linux/skbuff_ref.h +++ b/include/linux/skbuff_ref.h @@ -17,7 +17,7 @@ */ static inline void __skb_frag_ref(skb_frag_t *frag) { - get_page(skb_frag_page(frag)); + get_netmem(skb_frag_netmem(frag)); } /** @@ -40,7 +40,7 @@ static inline void skb_page_unref(netmem_ref netmem, bool recycle) if (recycle && napi_pp_put_page(netmem)) return; #endif - put_page(netmem_to_page(netmem)); + put_netmem(netmem); } /** diff --git a/include/net/netmem.h b/include/net/netmem.h index 973fdbcfef38b..ecb6b29c93f61 100644 --- a/include/net/netmem.h +++ b/include/net/netmem.h @@ -273,4 +273,7 @@ static inline unsigned long netmem_get_dma_addr(netmem_ref netmem) return __netmem_clear_lsb(netmem)->dma_addr; } +void get_netmem(netmem_ref netmem); +void put_netmem(netmem_ref netmem); + #endif /* _NET_NETMEM_H */ diff --git a/net/core/devmem.c b/net/core/devmem.c index f5c3a7e6dbb7b..dca2ff7cf6923 100644 --- a/net/core/devmem.c +++ b/net/core/devmem.c @@ -295,6 +295,16 @@ net_devmem_bind_dmabuf(struct net_device *dev, unsigned int dmabuf_fd, return ERR_PTR(err); } +void net_devmem_get_net_iov(struct net_iov *niov) +{ + net_devmem_dmabuf_binding_get(net_devmem_iov_binding(niov)); +} + +void net_devmem_put_net_iov(struct net_iov *niov) +{ + net_devmem_dmabuf_binding_put(net_devmem_iov_binding(niov)); +} + /*** "Dmabuf devmem memory provider" ***/ int mp_dmabuf_devmem_init(struct page_pool *pool) diff --git a/net/core/devmem.h b/net/core/devmem.h index 7fc158d527293..946f2e0157467 100644 --- a/net/core/devmem.h +++ b/net/core/devmem.h @@ -29,6 +29,10 @@ struct net_devmem_dmabuf_binding { * The binding undos itself and unmaps the underlying dmabuf once all * those refs are dropped and the binding is no longer desired or in * use. + * + * net_devmem_get_net_iov() on dmabuf net_iovs will increment this + * reference, making sure that the binding remains alive until all the + * net_iovs are no longer used. */ refcount_t ref; @@ -111,6 +115,9 @@ net_devmem_dmabuf_binding_put(struct net_devmem_dmabuf_binding *binding) __net_devmem_dmabuf_binding_free(binding); } +void net_devmem_get_net_iov(struct net_iov *niov); +void net_devmem_put_net_iov(struct net_iov *niov); + struct net_iov * net_devmem_alloc_dmabuf(struct net_devmem_dmabuf_binding *binding); void net_devmem_free_dmabuf(struct net_iov *ppiov); @@ -120,6 +127,19 @@ bool net_is_devmem_iov(struct net_iov *niov); #else struct net_devmem_dmabuf_binding; +static inline void +net_devmem_dmabuf_binding_put(struct net_devmem_dmabuf_binding *binding) +{ +} + +static inline void net_devmem_get_net_iov(struct net_iov *niov) +{ +} + +static inline void net_devmem_put_net_iov(struct net_iov *niov) +{ +} + static inline void __net_devmem_dmabuf_binding_free(struct net_devmem_dmabuf_binding *binding) { diff --git a/net/core/skbuff.c b/net/core/skbuff.c index d73ad79fe739d..00c22bce98e44 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -89,6 +89,7 @@ #include #include "dev.h" +#include "devmem.h" #include "netmem_priv.h" #include "sock_destructor.h" @@ -7313,3 +7314,32 @@ bool csum_and_copy_from_iter_full(void *addr, size_t bytes, return false; } EXPORT_SYMBOL(csum_and_copy_from_iter_full); + +void get_netmem(netmem_ref netmem) +{ + struct net_iov *niov; + + if (netmem_is_net_iov(netmem)) { + niov = netmem_to_net_iov(netmem); + if (net_is_devmem_iov(niov)) + net_devmem_get_net_iov(netmem_to_net_iov(netmem)); + return; + } + get_page(netmem_to_page(netmem)); +} +EXPORT_SYMBOL(get_netmem); + +void put_netmem(netmem_ref netmem) +{ + struct net_iov *niov; + + if (netmem_is_net_iov(netmem)) { + niov = netmem_to_net_iov(netmem); + if (net_is_devmem_iov(niov)) + net_devmem_put_net_iov(netmem_to_net_iov(netmem)); + return; + } + + put_page(netmem_to_page(netmem)); +} +EXPORT_SYMBOL(put_netmem);