From patchwork Tue Jan 19 15:36:48 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= X-Patchwork-Id: 366722 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C0D6AC433E0 for ; Tue, 19 Jan 2021 15:39:05 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 83ACC23133 for ; Tue, 19 Jan 2021 15:39:04 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2391585AbhASPim (ORCPT ); Tue, 19 Jan 2021 10:38:42 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39132 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2390832AbhASPhu (ORCPT ); Tue, 19 Jan 2021 10:37:50 -0500 Received: from mail-lf1-x12d.google.com (mail-lf1-x12d.google.com [IPv6:2a00:1450:4864:20::12d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6BC42C061575; Tue, 19 Jan 2021 07:37:10 -0800 (PST) Received: by mail-lf1-x12d.google.com with SMTP id h7so415505lfc.6; Tue, 19 Jan 2021 07:37:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=i4w3bZuLfrJkmf9BkM/u7+HKCeUtJU84EklOrB2XMPI=; b=PBajJlrkZrjHFsq2TdWP88VM9M/OVzk2ypQu20/Q0wMgELBNkEPDI3EQORNIvF74PU kIFt47jqc8RfODLoniwrZHYtKU/dKDkpuqWG2lPgZHCWwSwGPg9rOq70FR64lLpo+tSz 2npC7/husCLVPtuUv394avV9vP9oYfil4y4KjEQIQijVnVHgbl3wrcSFzsdmT3+pf41j iCxaJ0OMJ4z6enZZ/FkxPZ0Fy8/hNLZLZoKmoNRWxC2/FzQgM507GvqPNbmzgEFNH3fn U8xtSLQVi1i5/YRITDdPcnibS3zvI53+qT27VJ3S6UhITUUwpiejiJ1FDOjHXMYV8Mn4 X25g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=i4w3bZuLfrJkmf9BkM/u7+HKCeUtJU84EklOrB2XMPI=; b=ejMgLOdjXh3Xt1UbJ0V53nHDJWfvOoOmIzI0frp7ualOD5/RJLWjCV3pFup/2+A13p PTawzNHYC7MY4QqDIfnSoENiPp58fGMzRHyHqgT2Y19PzHKb2D7l7BHL7wtRzDCbcGgO UUZJeBpOCPSni/JKn/GINle+K9V8Wj6D+PxbW9V2JFc43p8AlNOxg3i0XUK2dBOwwi5V NEmUxtatheBIllyKoILn/Ptd8WdAKzHVyC6/1GGuUx5wUdOh2wV9NAQETRPGOpYuIC8y MPEukyxd0lI0fK9qWiJbTgvsCT41KLqAOcaARlljHMqRyCEM61JZyER6fly9tT84nBnI ZeGw== X-Gm-Message-State: AOAM5307oJwvN/KDPlDGclU4tD2WshCMIH1D0VG2sH4aNrCmIDPhcvck u3NqAj0mKjD2KrnTf6jpnylO+BAFML83xQ== X-Google-Smtp-Source: ABdhPJyefg83nhT+5S3clkof/5UuzOFgxk+ITucEPfi+MoRlLJWyAP6ks14pAh13bph3hzyv4qzRPA== X-Received: by 2002:ac2:58ee:: with SMTP id v14mr2135737lfo.298.1611070628823; Tue, 19 Jan 2021 07:37:08 -0800 (PST) Received: from btopel-mobl.ger.intel.com (c213-102-90-208.bredband.comhem.se. [213.102.90.208]) by smtp.gmail.com with ESMTPSA id z2sm2309075lfd.142.2021.01.19.07.37.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 19 Jan 2021 07:37:07 -0800 (PST) From: =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= To: ast@kernel.org, daniel@iogearbox.net, netdev@vger.kernel.org, bpf@vger.kernel.org Cc: =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= , magnus.karlsson@intel.com, maciej.fijalkowski@intel.com, kuba@kernel.org, jonathan.lemon@gmail.com, maximmi@nvidia.com, davem@davemloft.net, hawk@kernel.org, john.fastabend@gmail.com, ciara.loftus@intel.com, weqaar.a.janjua@intel.com Subject: [PATCH bpf-next 1/8] xdp: restructure redirect actions Date: Tue, 19 Jan 2021 16:36:48 +0100 Message-Id: <20210119153655.153999-2-bjorn.topel@gmail.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20210119153655.153999-1-bjorn.topel@gmail.com> References: <20210119153655.153999-1-bjorn.topel@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Björn Töpel The XDP_REDIRECT implementations for maps and non-maps are fairly similar, but obviously need to take different code paths depending on if the target is using a map or not. Today, the redirect targets for XDP either uses a map, or is based on ifindex. Future commits will introduce yet another redirect target via the a new helper, bpf_redirect_xsk(). To pave the way for that, we introduce an explicit redirect type to bpf_redirect_info. This makes the code easier to follow, and makes it easier to add new redirect targets. Further, using an explicit type in bpf_redirect_info has a slight positive performance impact by avoiding a pointer indirection for the map type lookup, and instead use the hot cacheline for bpf_redirect_info. The bpf_redirect_info flags member is not used by XDP, and not read/written any more. The map member is only written to when required/used, and not unconditionally. Reviewed-by: Maciej Fijalkowski Signed-off-by: Björn Töpel --- include/linux/filter.h | 9 ++ include/trace/events/xdp.h | 46 +++++++---- net/core/filter.c | 164 ++++++++++++++++++------------------- 3 files changed, 117 insertions(+), 102 deletions(-) diff --git a/include/linux/filter.h b/include/linux/filter.h index 7fdce5407214..5fc336a271c2 100644 --- a/include/linux/filter.h +++ b/include/linux/filter.h @@ -637,10 +637,19 @@ struct bpf_redirect_info { u32 tgt_index; void *tgt_value; struct bpf_map *map; + u32 tgt_type; u32 kern_flags; struct bpf_nh_params nh; }; +enum xdp_redirect_type { + XDP_REDIR_UNSET, + XDP_REDIR_DEV_IFINDEX, + XDP_REDIR_DEV_MAP, + XDP_REDIR_CPU_MAP, + XDP_REDIR_XSK_MAP, +}; + DECLARE_PER_CPU(struct bpf_redirect_info, bpf_redirect_info); /* flags for bpf_redirect_info kern_flags */ diff --git a/include/trace/events/xdp.h b/include/trace/events/xdp.h index 76a97176ab81..0e17b9a74f28 100644 --- a/include/trace/events/xdp.h +++ b/include/trace/events/xdp.h @@ -96,9 +96,10 @@ DECLARE_EVENT_CLASS(xdp_redirect_template, TP_PROTO(const struct net_device *dev, const struct bpf_prog *xdp, const void *tgt, int err, - const struct bpf_map *map, u32 index), + enum xdp_redirect_type type, + const struct bpf_redirect_info *ri), - TP_ARGS(dev, xdp, tgt, err, map, index), + TP_ARGS(dev, xdp, tgt, err, type, ri), TP_STRUCT__entry( __field(int, prog_id) @@ -111,12 +112,19 @@ DECLARE_EVENT_CLASS(xdp_redirect_template, ), TP_fast_assign( + struct bpf_map *map = NULL; + u32 index = ri->tgt_index; + + if (type == XDP_REDIR_DEV_MAP || type == XDP_REDIR_CPU_MAP || + type == XDP_REDIR_XSK_MAP) + map = READ_ONCE(ri->map); + __entry->prog_id = xdp->aux->id; __entry->act = XDP_REDIRECT; __entry->ifindex = dev->ifindex; __entry->err = err; __entry->to_ifindex = map ? devmap_ifindex(tgt, map) : - index; + (u32)(long)tgt; __entry->map_id = map ? map->id : 0; __entry->map_index = map ? index : 0; ), @@ -133,45 +141,49 @@ DEFINE_EVENT(xdp_redirect_template, xdp_redirect, TP_PROTO(const struct net_device *dev, const struct bpf_prog *xdp, const void *tgt, int err, - const struct bpf_map *map, u32 index), - TP_ARGS(dev, xdp, tgt, err, map, index) + enum xdp_redirect_type type, + const struct bpf_redirect_info *ri), + TP_ARGS(dev, xdp, tgt, err, type, ri) ); DEFINE_EVENT(xdp_redirect_template, xdp_redirect_err, TP_PROTO(const struct net_device *dev, const struct bpf_prog *xdp, const void *tgt, int err, - const struct bpf_map *map, u32 index), - TP_ARGS(dev, xdp, tgt, err, map, index) + enum xdp_redirect_type type, + const struct bpf_redirect_info *ri), + TP_ARGS(dev, xdp, tgt, err, type, ri) ); #define _trace_xdp_redirect(dev, xdp, to) \ - trace_xdp_redirect(dev, xdp, NULL, 0, NULL, to) + trace_xdp_redirect(dev, xdp, NULL, 0, XDP_REDIR_DEV_IFINDEX, NULL) #define _trace_xdp_redirect_err(dev, xdp, to, err) \ - trace_xdp_redirect_err(dev, xdp, NULL, err, NULL, to) + trace_xdp_redirect_err(dev, xdp, NULL, err, XDP_REDIR_DEV_IFINDEX, NULL) -#define _trace_xdp_redirect_map(dev, xdp, to, map, index) \ - trace_xdp_redirect(dev, xdp, to, 0, map, index) +#define _trace_xdp_redirect_map(dev, xdp, to, type, ri) \ + trace_xdp_redirect(dev, xdp, to, 0, type, ri) -#define _trace_xdp_redirect_map_err(dev, xdp, to, map, index, err) \ - trace_xdp_redirect_err(dev, xdp, to, err, map, index) +#define _trace_xdp_redirect_map_err(dev, xdp, to, type, ri, err) \ + trace_xdp_redirect_err(dev, xdp, to, err, type, ri) /* not used anymore, but kept around so as not to break old programs */ DEFINE_EVENT(xdp_redirect_template, xdp_redirect_map, TP_PROTO(const struct net_device *dev, const struct bpf_prog *xdp, const void *tgt, int err, - const struct bpf_map *map, u32 index), - TP_ARGS(dev, xdp, tgt, err, map, index) + enum xdp_redirect_type type, + const struct bpf_redirect_info *ri), + TP_ARGS(dev, xdp, tgt, err, type, ri) ); DEFINE_EVENT(xdp_redirect_template, xdp_redirect_map_err, TP_PROTO(const struct net_device *dev, const struct bpf_prog *xdp, const void *tgt, int err, - const struct bpf_map *map, u32 index), - TP_ARGS(dev, xdp, tgt, err, map, index) + enum xdp_redirect_type type, + const struct bpf_redirect_info *ri), + TP_ARGS(dev, xdp, tgt, err, type, ri) ); TRACE_EVENT(xdp_cpumap_kthread, diff --git a/net/core/filter.c b/net/core/filter.c index 9ab94e90d660..5f31e21be531 100644 --- a/net/core/filter.c +++ b/net/core/filter.c @@ -3923,23 +3923,6 @@ static const struct bpf_func_proto bpf_xdp_adjust_meta_proto = { .arg2_type = ARG_ANYTHING, }; -static int __bpf_tx_xdp_map(struct net_device *dev_rx, void *fwd, - struct bpf_map *map, struct xdp_buff *xdp) -{ - switch (map->map_type) { - case BPF_MAP_TYPE_DEVMAP: - case BPF_MAP_TYPE_DEVMAP_HASH: - return dev_map_enqueue(fwd, xdp, dev_rx); - case BPF_MAP_TYPE_CPUMAP: - return cpu_map_enqueue(fwd, xdp, dev_rx); - case BPF_MAP_TYPE_XSKMAP: - return __xsk_map_redirect(fwd, xdp); - default: - return -EBADRQC; - } - return 0; -} - void xdp_do_flush(void) { __dev_flush(); @@ -3948,22 +3931,6 @@ void xdp_do_flush(void) } EXPORT_SYMBOL_GPL(xdp_do_flush); -static inline void *__xdp_map_lookup_elem(struct bpf_map *map, u32 index) -{ - switch (map->map_type) { - case BPF_MAP_TYPE_DEVMAP: - return __dev_map_lookup_elem(map, index); - case BPF_MAP_TYPE_DEVMAP_HASH: - return __dev_map_hash_lookup_elem(map, index); - case BPF_MAP_TYPE_CPUMAP: - return __cpu_map_lookup_elem(map, index); - case BPF_MAP_TYPE_XSKMAP: - return __xsk_map_lookup_elem(map, index); - default: - return NULL; - } -} - void bpf_clear_redirect_map(struct bpf_map *map) { struct bpf_redirect_info *ri; @@ -3985,34 +3952,42 @@ int xdp_do_redirect(struct net_device *dev, struct xdp_buff *xdp, struct bpf_prog *xdp_prog) { struct bpf_redirect_info *ri = this_cpu_ptr(&bpf_redirect_info); - struct bpf_map *map = READ_ONCE(ri->map); - u32 index = ri->tgt_index; + enum xdp_redirect_type type = ri->tgt_type; void *fwd = ri->tgt_value; int err; - ri->tgt_index = 0; + ri->tgt_type = XDP_REDIR_UNSET; ri->tgt_value = NULL; - WRITE_ONCE(ri->map, NULL); - if (unlikely(!map)) { - fwd = dev_get_by_index_rcu(dev_net(dev), index); + switch (type) { + case XDP_REDIR_DEV_IFINDEX: + fwd = dev_get_by_index_rcu(dev_net(dev), (u32)(long)fwd); if (unlikely(!fwd)) { err = -EINVAL; - goto err; + break; } - err = dev_xdp_enqueue(fwd, xdp, dev); - } else { - err = __bpf_tx_xdp_map(dev, fwd, map, xdp); + break; + case XDP_REDIR_DEV_MAP: + err = dev_map_enqueue(fwd, xdp, dev); + break; + case XDP_REDIR_CPU_MAP: + err = cpu_map_enqueue(fwd, xdp, dev); + break; + case XDP_REDIR_XSK_MAP: + err = __xsk_map_redirect(fwd, xdp); + break; + default: + err = -EBADRQC; } if (unlikely(err)) goto err; - _trace_xdp_redirect_map(dev, xdp_prog, fwd, map, index); + _trace_xdp_redirect_map(dev, xdp_prog, fwd, type, ri); return 0; err: - _trace_xdp_redirect_map_err(dev, xdp_prog, fwd, map, index, err); + _trace_xdp_redirect_map_err(dev, xdp_prog, fwd, type, ri, err); return err; } EXPORT_SYMBOL_GPL(xdp_do_redirect); @@ -4021,41 +3996,40 @@ static int xdp_do_generic_redirect_map(struct net_device *dev, struct sk_buff *skb, struct xdp_buff *xdp, struct bpf_prog *xdp_prog, - struct bpf_map *map) + void *fwd, + enum xdp_redirect_type type) { struct bpf_redirect_info *ri = this_cpu_ptr(&bpf_redirect_info); - u32 index = ri->tgt_index; - void *fwd = ri->tgt_value; - int err = 0; - - ri->tgt_index = 0; - ri->tgt_value = NULL; - WRITE_ONCE(ri->map, NULL); + int err; - if (map->map_type == BPF_MAP_TYPE_DEVMAP || - map->map_type == BPF_MAP_TYPE_DEVMAP_HASH) { + switch (type) { + case XDP_REDIR_DEV_MAP: { struct bpf_dtab_netdev *dst = fwd; err = dev_map_generic_redirect(dst, skb, xdp_prog); if (unlikely(err)) goto err; - } else if (map->map_type == BPF_MAP_TYPE_XSKMAP) { + break; + } + case XDP_REDIR_XSK_MAP: { struct xdp_sock *xs = fwd; err = xsk_generic_rcv(xs, xdp); if (err) goto err; consume_skb(skb); - } else { + break; + } + default: /* TODO: Handle BPF_MAP_TYPE_CPUMAP */ err = -EBADRQC; goto err; } - _trace_xdp_redirect_map(dev, xdp_prog, fwd, map, index); + _trace_xdp_redirect_map(dev, xdp_prog, fwd, type, ri); return 0; err: - _trace_xdp_redirect_map_err(dev, xdp_prog, fwd, map, index, err); + _trace_xdp_redirect_map_err(dev, xdp_prog, fwd, type, ri, err); return err; } @@ -4063,29 +4037,31 @@ int xdp_do_generic_redirect(struct net_device *dev, struct sk_buff *skb, struct xdp_buff *xdp, struct bpf_prog *xdp_prog) { struct bpf_redirect_info *ri = this_cpu_ptr(&bpf_redirect_info); - struct bpf_map *map = READ_ONCE(ri->map); - u32 index = ri->tgt_index; - struct net_device *fwd; + enum xdp_redirect_type type = ri->tgt_type; + void *fwd = ri->tgt_value; int err = 0; - if (map) - return xdp_do_generic_redirect_map(dev, skb, xdp, xdp_prog, - map); - ri->tgt_index = 0; - fwd = dev_get_by_index_rcu(dev_net(dev), index); - if (unlikely(!fwd)) { - err = -EINVAL; - goto err; - } + ri->tgt_type = XDP_REDIR_UNSET; + ri->tgt_value = NULL; - err = xdp_ok_fwd_dev(fwd, skb->len); - if (unlikely(err)) - goto err; + if (type == XDP_REDIR_DEV_IFINDEX) { + fwd = dev_get_by_index_rcu(dev_net(dev), (u32)(long)fwd); + if (unlikely(!fwd)) { + err = -EINVAL; + goto err; + } - skb->dev = fwd; - _trace_xdp_redirect(dev, xdp_prog, index); - generic_xdp_tx(skb, xdp_prog); - return 0; + err = xdp_ok_fwd_dev(fwd, skb->len); + if (unlikely(err)) + goto err; + + skb->dev = fwd; + _trace_xdp_redirect(dev, xdp_prog, index); + generic_xdp_tx(skb, xdp_prog); + return 0; + } + + return xdp_do_generic_redirect_map(dev, skb, xdp, xdp_prog, fwd, type); err: _trace_xdp_redirect_err(dev, xdp_prog, index, err); return err; @@ -4098,10 +4074,9 @@ BPF_CALL_2(bpf_xdp_redirect, u32, ifindex, u64, flags) if (unlikely(flags)) return XDP_ABORTED; - ri->flags = flags; - ri->tgt_index = ifindex; - ri->tgt_value = NULL; - WRITE_ONCE(ri->map, NULL); + ri->tgt_type = XDP_REDIR_DEV_IFINDEX; + ri->tgt_index = 0; + ri->tgt_value = (void *)(long)ifindex; return XDP_REDIRECT; } @@ -4123,18 +4098,37 @@ BPF_CALL_3(bpf_xdp_redirect_map, struct bpf_map *, map, u32, ifindex, if (unlikely(flags > XDP_TX)) return XDP_ABORTED; - ri->tgt_value = __xdp_map_lookup_elem(map, ifindex); + switch (map->map_type) { + case BPF_MAP_TYPE_DEVMAP: + ri->tgt_value = __dev_map_lookup_elem(map, ifindex); + ri->tgt_type = XDP_REDIR_DEV_MAP; + break; + case BPF_MAP_TYPE_DEVMAP_HASH: + ri->tgt_value = __dev_map_hash_lookup_elem(map, ifindex); + ri->tgt_type = XDP_REDIR_DEV_MAP; + break; + case BPF_MAP_TYPE_CPUMAP: + ri->tgt_value = __cpu_map_lookup_elem(map, ifindex); + ri->tgt_type = XDP_REDIR_CPU_MAP; + break; + case BPF_MAP_TYPE_XSKMAP: + ri->tgt_value = __xsk_map_lookup_elem(map, ifindex); + ri->tgt_type = XDP_REDIR_XSK_MAP; + break; + default: + ri->tgt_value = NULL; + } + if (unlikely(!ri->tgt_value)) { /* If the lookup fails we want to clear out the state in the * redirect_info struct completely, so that if an eBPF program * performs multiple lookups, the last one always takes * precedence. */ - WRITE_ONCE(ri->map, NULL); + ri->tgt_type = XDP_REDIR_UNSET; return flags; } - ri->flags = flags; ri->tgt_index = ifindex; WRITE_ONCE(ri->map, map); From patchwork Tue Jan 19 15:36:49 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= X-Patchwork-Id: 366719 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E96F6C433E0 for ; Tue, 19 Jan 2021 15:41:56 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id BF78623135 for ; Tue, 19 Jan 2021 15:41:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2390490AbhASPlk (ORCPT ); Tue, 19 Jan 2021 10:41:40 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39144 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2391102AbhASPhy (ORCPT ); Tue, 19 Jan 2021 10:37:54 -0500 Received: from mail-lj1-x229.google.com (mail-lj1-x229.google.com [IPv6:2a00:1450:4864:20::229]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6BF84C061757; Tue, 19 Jan 2021 07:37:13 -0800 (PST) Received: by mail-lj1-x229.google.com with SMTP id m13so22360485ljo.11; Tue, 19 Jan 2021 07:37:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=RO1GdsGe/CwUlFc4QGSwFYrVwnR9WPHVNLK3C1RF8Gk=; b=nl/1LuKal+LY2hUK70DvHFUEAVo/Y8S9ZeNS34cNf82aMCQoX5Kd+xuLUDnf8jjSzu ICZf2EL6pykZWxQ2r4YcOML7rut3mV35IGAr+hvvYrKiNjxc+4AKmfvpdeb6RZUKcFG2 dwVajnLd0dngeQrBBSC/nj6yUun992iJadbK6yrk0pngL5qOBKW0U3CRc7u1eRLYeeW2 k8aosAWp4CT9EWmi9MsUz7yXvWYHvGN3SzI1kgP4jYlmDcAJBGk9LbZRDj+iCeADw4x4 7/Cyz5oJK56XpdHnCf+hcNad3sVCYyRCmVyxMYBkWPXhM3atRE8lt7F3pY36HpXbHcMX suCw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=RO1GdsGe/CwUlFc4QGSwFYrVwnR9WPHVNLK3C1RF8Gk=; b=RpAwYlNj0Io6DkOprb5AfzsRFJZqybAgaMXJWXPSM5DqOE9gYpK312lUx9QVT+KNrH uSBxZAVz/KKPUN8bgXHvP6fGRHdUaXjKLScLA4jyJGNruP9WDyx1sGO7FJzHaQag1IxT P7ZXW9oHovocWG834nLGSfoh+PWNqB2hJA75ZwkwCDQSAGTMuadsy3rI9exOu+96oeFf GgFJyUtfpMt07gxMTrkjb+A4poSc46hTL/XEGjZXmABsEAOb93hzXk2IGJfhfeGY/qVr 7AgUdvyfhwnssE4HcXhCObbN2Dl2qURQQi3XgYjxIIiCJBrJUibHecUiGOuYLro5ai3m Pg6g== X-Gm-Message-State: AOAM532XspBio1NxBIpfgB3U9Jw5+sKOHCVU4kTFQlZTIEWgsGWzBTDi q6PQYllaXp07sOaIuAZCWaQ= X-Google-Smtp-Source: ABdhPJxnJswRzGJUeUIiHcCMTkZmw2nYvj9Db6HS5/SCHwD81ZBQaxSqWpaXolCEf66+7EVATsEuiQ== X-Received: by 2002:a05:651c:1123:: with SMTP id e3mr2101985ljo.131.1611070630431; Tue, 19 Jan 2021 07:37:10 -0800 (PST) Received: from btopel-mobl.ger.intel.com (c213-102-90-208.bredband.comhem.se. [213.102.90.208]) by smtp.gmail.com with ESMTPSA id z2sm2309075lfd.142.2021.01.19.07.37.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 19 Jan 2021 07:37:09 -0800 (PST) From: =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= To: ast@kernel.org, daniel@iogearbox.net, netdev@vger.kernel.org, bpf@vger.kernel.org Cc: =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= , magnus.karlsson@intel.com, maciej.fijalkowski@intel.com, kuba@kernel.org, jonathan.lemon@gmail.com, maximmi@nvidia.com, davem@davemloft.net, hawk@kernel.org, john.fastabend@gmail.com, ciara.loftus@intel.com, weqaar.a.janjua@intel.com Subject: [PATCH bpf-next 2/8] xsk: remove explicit_free parameter from __xsk_rcv() Date: Tue, 19 Jan 2021 16:36:49 +0100 Message-Id: <20210119153655.153999-3-bjorn.topel@gmail.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20210119153655.153999-1-bjorn.topel@gmail.com> References: <20210119153655.153999-1-bjorn.topel@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Björn Töpel The explicit_free parameter of the __xsk_rcv() function was used to mark whether the call was via the generic XDP or the native XDP path. Instead of clutter the code with if-statements and "true/false" parameters which are hard to understand, simply move the explicit free to the __xsk_map_redirect() which is always called from the native XDP path. Reviewed-by: Maciej Fijalkowski Signed-off-by: Björn Töpel --- net/xdp/xsk.c | 47 +++++++++++++++++++++++++++++++---------------- 1 file changed, 31 insertions(+), 16 deletions(-) diff --git a/net/xdp/xsk.c b/net/xdp/xsk.c index 8037b04a9edd..5820de65060b 100644 --- a/net/xdp/xsk.c +++ b/net/xdp/xsk.c @@ -184,12 +184,13 @@ static void xsk_copy_xdp(struct xdp_buff *to, struct xdp_buff *from, u32 len) memcpy(to_buf, from_buf, len + metalen); } -static int __xsk_rcv(struct xdp_sock *xs, struct xdp_buff *xdp, u32 len, - bool explicit_free) +static int __xsk_rcv(struct xdp_sock *xs, struct xdp_buff *xdp) { struct xdp_buff *xsk_xdp; int err; + u32 len; + len = xdp->data_end - xdp->data; if (len > xsk_pool_get_rx_frame_size(xs->pool)) { xs->rx_dropped++; return -ENOSPC; @@ -207,8 +208,6 @@ static int __xsk_rcv(struct xdp_sock *xs, struct xdp_buff *xdp, u32 len, xsk_buff_free(xsk_xdp); return err; } - if (explicit_free) - xdp_return_buff(xdp); return 0; } @@ -230,11 +229,8 @@ static bool xsk_is_bound(struct xdp_sock *xs) return false; } -static int xsk_rcv(struct xdp_sock *xs, struct xdp_buff *xdp, - bool explicit_free) +static int xsk_rcv_check(struct xdp_sock *xs, struct xdp_buff *xdp) { - u32 len; - if (!xsk_is_bound(xs)) return -EINVAL; @@ -242,11 +238,7 @@ static int xsk_rcv(struct xdp_sock *xs, struct xdp_buff *xdp, return -EINVAL; sk_mark_napi_id_once_xdp(&xs->sk, xdp); - len = xdp->data_end - xdp->data; - - return xdp->rxq->mem.type == MEM_TYPE_XSK_BUFF_POOL ? - __xsk_rcv_zc(xs, xdp, len) : - __xsk_rcv(xs, xdp, len, explicit_free); + return 0; } static void xsk_flush(struct xdp_sock *xs) @@ -261,18 +253,41 @@ int xsk_generic_rcv(struct xdp_sock *xs, struct xdp_buff *xdp) int err; spin_lock_bh(&xs->rx_lock); - err = xsk_rcv(xs, xdp, false); - xsk_flush(xs); + err = xsk_rcv_check(xs, xdp); + if (!err) { + err = __xsk_rcv(xs, xdp); + xsk_flush(xs); + } spin_unlock_bh(&xs->rx_lock); return err; } +static int xsk_rcv(struct xdp_sock *xs, struct xdp_buff *xdp) +{ + int err; + u32 len; + + err = xsk_rcv_check(xs, xdp); + if (err) + return err; + + if (xdp->rxq->mem.type == MEM_TYPE_XSK_BUFF_POOL) { + len = xdp->data_end - xdp->data; + return __xsk_rcv_zc(xs, xdp, len); + } + + err = __xsk_rcv(xs, xdp); + if (!err) + xdp_return_buff(xdp); + return err; +} + int __xsk_map_redirect(struct xdp_sock *xs, struct xdp_buff *xdp) { struct list_head *flush_list = this_cpu_ptr(&xskmap_flush_list); int err; - err = xsk_rcv(xs, xdp, true); + err = xsk_rcv(xs, xdp); if (err) return err; From patchwork Tue Jan 19 15:36:52 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= X-Patchwork-Id: 366718 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7C6E3C433DB for ; Tue, 19 Jan 2021 15:42:33 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 41E2123135 for ; Tue, 19 Jan 2021 15:42:33 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2391673AbhASPmQ (ORCPT ); Tue, 19 Jan 2021 10:42:16 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39038 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2390660AbhASPhd (ORCPT ); Tue, 19 Jan 2021 10:37:33 -0500 Received: from mail-lf1-x136.google.com (mail-lf1-x136.google.com [IPv6:2a00:1450:4864:20::136]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 81BBAC0613ED; Tue, 19 Jan 2021 07:37:16 -0800 (PST) Received: by mail-lf1-x136.google.com with SMTP id o13so29680110lfr.3; Tue, 19 Jan 2021 07:37:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=/A8qSxg6mjQyW3rVEfH/72JcIK5m3UMA6EDeCF6zH2o=; b=a9+sJbO58jlhdcDYj/6WhFSwdgMZw3d1bu4Z5IOzn7x+hxlcTdKtkTshuYr4YR0sQB Y0bO9g8Ju88kT2ChMvkYC8SKrAFbmzFeN73cGFCM8S9Mt1GN8SIgQhxC0dATQ8wu1J5m Y2GhAc04bjp+Rb0Trlir4Jwr/bIbyPZHNlux7eze2zyehBJvX4bFk40zN7fgJYL/+cZh mQ45OS9rTQzQEwfeZbesVdMb4gPGn/K/yj1ojbJRF/26YtCFIg6biSFkP3/BTNHi4GlR kjiPy+WNl9Li95YlKL+mYmolOyvoJfo9yeP9QoWjRyUTfL6zjliVeIYKqU0WJzHFmh+T +J/A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=/A8qSxg6mjQyW3rVEfH/72JcIK5m3UMA6EDeCF6zH2o=; b=VRYszSGojOhGmJIKuJBmy/kJbYNKjXj8rmkKZnIkC99G1X8uqjud3L0JcAyNTAJt5r KqGtUXrGQr37qXvMiu/2s9/5nBGEZTy/O02+Ei5+aEPZ3g8uNhjx6JhwqkSLCcMjt2X9 4p5cz/09D36xDgow+dKRf6ANOMjb4TG6J1XR5wGOnSCykrYk/LB5ctDzFcAze2VFuCp6 AQluG5dRBx15KKf6jnxY3/tWU5fBQYGgqKVypWnCkWScbocja/55lM0rjcUiHEwFpy6l BKxis82L0pQLEPBNN3kPI01vE1mbujNi/CMeU9yIiW3XoX8aRit/dYzioSz1d6QjKBrf s4bA== X-Gm-Message-State: AOAM530anAlU4pN4a/FhuVNjKnquCxRLez6qs+95W7jgFuh7Q/xZuWB+ Or/jf/X8ODUgYPmuHUxzd/E= X-Google-Smtp-Source: ABdhPJwM+lS45bj5hNo9KN/pO4S49aGEz+iBRNP0Bm74IFSe1KWkkw9ryeD4y3ySS+UCcJQJT4KazA== X-Received: by 2002:a19:6d0b:: with SMTP id i11mr1599898lfc.332.1611070635075; Tue, 19 Jan 2021 07:37:15 -0800 (PST) Received: from btopel-mobl.ger.intel.com (c213-102-90-208.bredband.comhem.se. [213.102.90.208]) by smtp.gmail.com with ESMTPSA id z2sm2309075lfd.142.2021.01.19.07.37.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 19 Jan 2021 07:37:14 -0800 (PST) From: =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= To: ast@kernel.org, daniel@iogearbox.net, netdev@vger.kernel.org, bpf@vger.kernel.org Cc: =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= , magnus.karlsson@intel.com, maciej.fijalkowski@intel.com, kuba@kernel.org, jonathan.lemon@gmail.com, maximmi@nvidia.com, davem@davemloft.net, hawk@kernel.org, john.fastabend@gmail.com, ciara.loftus@intel.com, weqaar.a.janjua@intel.com, Marek Majtyka Subject: [PATCH bpf-next 5/8] libbpf, xsk: select AF_XDP BPF program based on kernel version Date: Tue, 19 Jan 2021 16:36:52 +0100 Message-Id: <20210119153655.153999-6-bjorn.topel@gmail.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20210119153655.153999-1-bjorn.topel@gmail.com> References: <20210119153655.153999-1-bjorn.topel@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Björn Töpel Add detection for kernel version, and adapt the BPF program based on kernel support. This way, users will get the best possible performance from the BPF program. Reviewed-by: Maciej Fijalkowski Acked-by: Maciej Fijalkowski Signed-off-by: Björn Töpel Signed-off-by: Marek Majtyka --- tools/lib/bpf/libbpf.c | 2 +- tools/lib/bpf/libbpf_internal.h | 2 ++ tools/lib/bpf/libbpf_probes.c | 16 ------------- tools/lib/bpf/xsk.c | 41 ++++++++++++++++++++++++++++++--- 4 files changed, 41 insertions(+), 20 deletions(-) diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c index 2abbc3800568..6a53adf14a9c 100644 --- a/tools/lib/bpf/libbpf.c +++ b/tools/lib/bpf/libbpf.c @@ -693,7 +693,7 @@ bpf_object__add_programs(struct bpf_object *obj, Elf_Data *sec_data, return 0; } -static __u32 get_kernel_version(void) +__u32 get_kernel_version(void) { __u32 major, minor, patch; struct utsname info; diff --git a/tools/lib/bpf/libbpf_internal.h b/tools/lib/bpf/libbpf_internal.h index 969d0ac592ba..dafb780e2dd2 100644 --- a/tools/lib/bpf/libbpf_internal.h +++ b/tools/lib/bpf/libbpf_internal.h @@ -349,4 +349,6 @@ struct bpf_core_relo { enum bpf_core_relo_kind kind; }; +__u32 get_kernel_version(void); + #endif /* __LIBBPF_LIBBPF_INTERNAL_H */ diff --git a/tools/lib/bpf/libbpf_probes.c b/tools/lib/bpf/libbpf_probes.c index ecaae2927ab8..aae0231371d0 100644 --- a/tools/lib/bpf/libbpf_probes.c +++ b/tools/lib/bpf/libbpf_probes.c @@ -48,22 +48,6 @@ static int get_vendor_id(int ifindex) return strtol(buf, NULL, 0); } -static int get_kernel_version(void) -{ - int version, subversion, patchlevel; - struct utsname utsn; - - /* Return 0 on failure, and attempt to probe with empty kversion */ - if (uname(&utsn)) - return 0; - - if (sscanf(utsn.release, "%d.%d.%d", - &version, &subversion, &patchlevel) != 3) - return 0; - - return (version << 16) + (subversion << 8) + patchlevel; -} - static void probe_load(enum bpf_prog_type prog_type, const struct bpf_insn *insns, size_t insns_cnt, char *buf, size_t buf_len, __u32 ifindex) diff --git a/tools/lib/bpf/xsk.c b/tools/lib/bpf/xsk.c index e3e41ceeb1bc..c8642c6cb5d6 100644 --- a/tools/lib/bpf/xsk.c +++ b/tools/lib/bpf/xsk.c @@ -23,6 +23,7 @@ #include #include #include +#include #include #include #include @@ -46,6 +47,11 @@ #define PF_XDP AF_XDP #endif +enum xsk_prog { + XSK_PROG_FALLBACK, + XSK_PROG_REDIRECT_FLAGS, +}; + struct xsk_umem { struct xsk_ring_prod *fill_save; struct xsk_ring_cons *comp_save; @@ -351,6 +357,13 @@ int xsk_umem__create_v0_0_2(struct xsk_umem **umem_ptr, void *umem_area, COMPAT_VERSION(xsk_umem__create_v0_0_2, xsk_umem__create, LIBBPF_0.0.2) DEFAULT_VERSION(xsk_umem__create_v0_0_4, xsk_umem__create, LIBBPF_0.0.4) +static enum xsk_prog get_xsk_prog(void) +{ + __u32 kver = get_kernel_version(); + + return kver < KERNEL_VERSION(5, 3, 0) ? XSK_PROG_FALLBACK : XSK_PROG_REDIRECT_FLAGS; +} + static int xsk_load_xdp_prog(struct xsk_socket *xsk) { static const int log_buf_size = 16 * 1024; @@ -358,7 +371,7 @@ static int xsk_load_xdp_prog(struct xsk_socket *xsk) char log_buf[log_buf_size]; int err, prog_fd; - /* This is the C-program: + /* This is the fallback C-program: * SEC("xdp_sock") int xdp_sock_prog(struct xdp_md *ctx) * { * int ret, index = ctx->rx_queue_index; @@ -414,9 +427,31 @@ static int xsk_load_xdp_prog(struct xsk_socket *xsk) /* The jumps are to this instruction */ BPF_EXIT_INSN(), }; - size_t insns_cnt = sizeof(prog) / sizeof(struct bpf_insn); - prog_fd = bpf_load_program(BPF_PROG_TYPE_XDP, prog, insns_cnt, + /* This is the post-5.3 kernel C-program: + * SEC("xdp_sock") int xdp_sock_prog(struct xdp_md *ctx) + * { + * return bpf_redirect_map(&xsks_map, ctx->rx_queue_index, XDP_PASS); + * } + */ + struct bpf_insn prog_redirect_flags[] = { + /* r2 = *(u32 *)(r1 + 16) */ + BPF_LDX_MEM(BPF_W, BPF_REG_2, BPF_REG_1, 16), + /* r1 = xskmap[] */ + BPF_LD_MAP_FD(BPF_REG_1, ctx->xsks_map_fd), + /* r3 = XDP_PASS */ + BPF_MOV64_IMM(BPF_REG_3, 2), + /* call bpf_redirect_map */ + BPF_EMIT_CALL(BPF_FUNC_redirect_map), + BPF_EXIT_INSN(), + }; + size_t insns_cnt[] = {sizeof(prog) / sizeof(struct bpf_insn), + sizeof(prog_redirect_flags) / sizeof(struct bpf_insn), + }; + struct bpf_insn *progs[] = {prog, prog_redirect_flags}; + enum xsk_prog option = get_xsk_prog(); + + prog_fd = bpf_load_program(BPF_PROG_TYPE_XDP, progs[option], insns_cnt[option], "LGPL-2.1 or BSD-2-Clause", 0, log_buf, log_buf_size); if (prog_fd < 0) { From patchwork Tue Jan 19 15:36:53 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= X-Patchwork-Id: 366720 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D68F9C433E6 for ; Tue, 19 Jan 2021 15:40:25 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 90BD323134 for ; Tue, 19 Jan 2021 15:40:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2391604AbhASPjq (ORCPT ); Tue, 19 Jan 2021 10:39:46 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39166 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2391482AbhASPh6 (ORCPT ); Tue, 19 Jan 2021 10:37:58 -0500 Received: from mail-lf1-x131.google.com (mail-lf1-x131.google.com [IPv6:2a00:1450:4864:20::131]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 071AAC061786; Tue, 19 Jan 2021 07:37:18 -0800 (PST) Received: by mail-lf1-x131.google.com with SMTP id h205so29617064lfd.5; Tue, 19 Jan 2021 07:37:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=rXtdonoT6z2Mk/wuvXU5Af4iSxQJt1+pwiGmRinI7sw=; b=qRIv7JIkirHJdRcAWpGuGaN9AF/QwzHillhQDelsg+MdJxUdGCQtpjfFQyfraivpB/ rT0csC5/J5xVv20urpem6yD7g+ng+OPeBKuu1SJxlQvih6T3/Mp8C6NXJ4TOeQ+ADm8p BXSGyN1qVJHCsjpVvXv0o9jxOx/SyuVhGL6hcld5Nv/paIQy4+haZirfw48Rl+GcN6+5 4oCJsSwn6XroCkAp3Iix97jgVyJlGOn7gf8amq3YX5uvljQUY9dnxZfWUrxF98TkvsxX YU+4I/qyPPkrJNffDNFv4Ui6xz68wpa4QDDNZ4dPd+3KzRDmV6uLkwySLqZoLzEtMUfi B0UQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=rXtdonoT6z2Mk/wuvXU5Af4iSxQJt1+pwiGmRinI7sw=; b=ZUBaBTG4NCWNsO8g0h3/zlwBtu1cp9STv2g/BoVTiPup/rpe8wMz2xGvuz46ic1Ov1 dhzqUyOYKbEcNDL+qeWKjTzaDNC5ajvqYLr2mXwe2XMgbNQcEGH6imM3Lmdp74g5T5eP L7EwKjwgqDVLBIRDtq7Tckz5BE89Ko8Ni2lISdoyVO8ZAqvVQM4FsFCngqwe1yIZBnZ/ XwvsNlSYrGyzoMAefyQXQwWMPscGfeoSWqBZ1PFmcCMtMbad/pkRntQc7M5JLdWT/kek 6v/6O9KNt48Su49ALr8vmgYujVQbpvsdeHUg8vXzByszieYY2wb7IwPHKPTI5MrAxQS8 Ap+g== X-Gm-Message-State: AOAM533bIr4b7cbOMgZvarGRsKCnZbm/sNJ3ssZiXl3S8H9dJkilPz+x WKRQCRFPN9IcsR0JJD4w3lw= X-Google-Smtp-Source: ABdhPJy1FQUrnRBMebvcPkh02XkOq8Y2OFZutHujM6kkZReWtsYl8C35xBYOmiI2YxOlRdLom2rQgw== X-Received: by 2002:ac2:46dc:: with SMTP id p28mr202828lfo.25.1611070636553; Tue, 19 Jan 2021 07:37:16 -0800 (PST) Received: from btopel-mobl.ger.intel.com (c213-102-90-208.bredband.comhem.se. [213.102.90.208]) by smtp.gmail.com with ESMTPSA id z2sm2309075lfd.142.2021.01.19.07.37.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 19 Jan 2021 07:37:15 -0800 (PST) From: =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= To: ast@kernel.org, daniel@iogearbox.net, netdev@vger.kernel.org, bpf@vger.kernel.org Cc: =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= , magnus.karlsson@intel.com, maciej.fijalkowski@intel.com, kuba@kernel.org, jonathan.lemon@gmail.com, maximmi@nvidia.com, davem@davemloft.net, hawk@kernel.org, john.fastabend@gmail.com, ciara.loftus@intel.com, weqaar.a.janjua@intel.com Subject: [PATCH bpf-next 6/8] libbpf, xsk: select bpf_redirect_xsk(), if supported Date: Tue, 19 Jan 2021 16:36:53 +0100 Message-Id: <20210119153655.153999-7-bjorn.topel@gmail.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20210119153655.153999-1-bjorn.topel@gmail.com> References: <20210119153655.153999-1-bjorn.topel@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Björn Töpel Select bpf_redirect_xsk() as the default AF_XDP BPF program, if supported. The bpf_redirect_xsk() helper does not require an XSKMAP, so make sure that no map is created/updated when using it. Reviewed-by: Maciej Fijalkowski Signed-off-by: Björn Töpel --- tools/lib/bpf/xsk.c | 46 +++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 42 insertions(+), 4 deletions(-) diff --git a/tools/lib/bpf/xsk.c b/tools/lib/bpf/xsk.c index c8642c6cb5d6..27e36d6d92a6 100644 --- a/tools/lib/bpf/xsk.c +++ b/tools/lib/bpf/xsk.c @@ -47,9 +47,12 @@ #define PF_XDP AF_XDP #endif +#define XSKMAP_NOT_NEEDED -1 + enum xsk_prog { XSK_PROG_FALLBACK, XSK_PROG_REDIRECT_FLAGS, + XSK_PROG_REDIRECT_XSK, }; struct xsk_umem { @@ -361,7 +364,11 @@ static enum xsk_prog get_xsk_prog(void) { __u32 kver = get_kernel_version(); - return kver < KERNEL_VERSION(5, 3, 0) ? XSK_PROG_FALLBACK : XSK_PROG_REDIRECT_FLAGS; + if (kver < KERNEL_VERSION(5, 3, 0)) + return XSK_PROG_FALLBACK; + if (kver < KERNEL_VERSION(5, 12, 0)) + return XSK_PROG_REDIRECT_FLAGS; + return XSK_PROG_REDIRECT_XSK; } static int xsk_load_xdp_prog(struct xsk_socket *xsk) @@ -445,10 +452,25 @@ static int xsk_load_xdp_prog(struct xsk_socket *xsk) BPF_EMIT_CALL(BPF_FUNC_redirect_map), BPF_EXIT_INSN(), }; + + /* This is the post-5.12 kernel C-program: + * SEC("xdp_sock") int xdp_sock_prog(struct xdp_md *ctx) + * { + * return bpf_redirect_xsk(ctx, XDP_PASS); + * } + */ + struct bpf_insn prog_redirect_xsk[] = { + /* r2 = XDP_PASS */ + BPF_MOV64_IMM(BPF_REG_2, 2), + /* call bpf_redirect_xsk */ + BPF_EMIT_CALL(BPF_FUNC_redirect_xsk), + BPF_EXIT_INSN(), + }; size_t insns_cnt[] = {sizeof(prog) / sizeof(struct bpf_insn), sizeof(prog_redirect_flags) / sizeof(struct bpf_insn), + sizeof(prog_redirect_xsk) / sizeof(struct bpf_insn), }; - struct bpf_insn *progs[] = {prog, prog_redirect_flags}; + struct bpf_insn *progs[] = {prog, prog_redirect_flags, prog_redirect_xsk}; enum xsk_prog option = get_xsk_prog(); prog_fd = bpf_load_program(BPF_PROG_TYPE_XDP, progs[option], insns_cnt[option], @@ -508,12 +530,22 @@ static int xsk_get_max_queues(struct xsk_socket *xsk) return ret; } +static bool xskmap_required(void) +{ + return get_xsk_prog() != XSK_PROG_REDIRECT_XSK; +} + static int xsk_create_bpf_maps(struct xsk_socket *xsk) { struct xsk_ctx *ctx = xsk->ctx; int max_queues; int fd; + if (!xskmap_required()) { + ctx->xsks_map_fd = XSKMAP_NOT_NEEDED; + return 0; + } + max_queues = xsk_get_max_queues(xsk); if (max_queues < 0) return max_queues; @@ -532,6 +564,9 @@ static void xsk_delete_bpf_maps(struct xsk_socket *xsk) { struct xsk_ctx *ctx = xsk->ctx; + if (ctx->xsks_map_fd == XSKMAP_NOT_NEEDED) + return; + bpf_map_delete_elem(ctx->xsks_map_fd, &ctx->queue_id); close(ctx->xsks_map_fd); } @@ -563,7 +598,7 @@ static int xsk_lookup_bpf_maps(struct xsk_socket *xsk) if (err) goto out_map_ids; - ctx->xsks_map_fd = -1; + ctx->xsks_map_fd = XSKMAP_NOT_NEEDED; for (i = 0; i < prog_info.nr_map_ids; i++) { fd = bpf_map_get_fd_by_id(map_ids[i]); @@ -585,7 +620,7 @@ static int xsk_lookup_bpf_maps(struct xsk_socket *xsk) } err = 0; - if (ctx->xsks_map_fd == -1) + if (ctx->xsks_map_fd == XSKMAP_NOT_NEEDED && xskmap_required()) err = -ENOENT; out_map_ids: @@ -597,6 +632,9 @@ static int xsk_set_bpf_maps(struct xsk_socket *xsk) { struct xsk_ctx *ctx = xsk->ctx; + if (ctx->xsks_map_fd == XSKMAP_NOT_NEEDED) + return 0; + return bpf_map_update_elem(ctx->xsks_map_fd, &ctx->queue_id, &xsk->fd, 0); } From patchwork Tue Jan 19 15:36:54 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= X-Patchwork-Id: 366721 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0B1D8C433E6 for ; Tue, 19 Jan 2021 15:39:35 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id BE18C23135 for ; Tue, 19 Jan 2021 15:39:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2391590AbhASPjc (ORCPT ); Tue, 19 Jan 2021 10:39:32 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39172 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2391539AbhASPh7 (ORCPT ); Tue, 19 Jan 2021 10:37:59 -0500 Received: from mail-lf1-x132.google.com (mail-lf1-x132.google.com [IPv6:2a00:1450:4864:20::132]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 54836C061793; Tue, 19 Jan 2021 07:37:19 -0800 (PST) Received: by mail-lf1-x132.google.com with SMTP id m25so29621723lfc.11; Tue, 19 Jan 2021 07:37:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Mzur33W7H1Bgzwa0dFrLF2wAgaQwxYBaCzvYFoXPOHU=; b=g410lA8FMwC6f+P2pwDXfImkiS+xpXRZb2lhTLCq4pwZNz5zqgaOJBmLdBJK8ihmYV CsXAzUKB//54anGcEBLElRHiUgitWKot1t695762lgxjE8CeR2QYzBs8OR92R8pb/52x uVoW+zv6sOam23MnxArZUIRWB+SBWdCBak0ZoSfyha0rDKwtt6wtPMJhZOlVpNAzS/8s YzOADREppyUyhL1snNTF7tO/9d/ylVnnaR5HHepy79gszrrVTKDBkGJan3h+kFZjgLya f/j2ugexxtyTK35OZkbI4kcRz/Qu6etC2qNXFzEpsouiDzpyzqDBZsHBfXRz7t49wOwE WTfg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Mzur33W7H1Bgzwa0dFrLF2wAgaQwxYBaCzvYFoXPOHU=; b=MebriF7crs9yd8hodPVmcPpHtEjm2M5ZX0NnLe0s4sEEC61GGb2S/FPcSw39d5Y7bf /ZX2j9nG55crr3NVjRovbmZC8e3nTVI3jR4qRJxbVOuzevxYY1X09hyNv78Wrch+EK9a +mFNVGlltlJHaKtSo2oAp1aMLQsq1lTT13zSmuvQVBv2L/CC0oZk6PTZUJspyJhFOUVf MHp/krGjoetwCjg8MdFuVR3HdVg3G/Tc3/3hqcpP2Gezo8ofh/fZ3Af43lrgNjhBsiTs JLHS348bMyCGNQ0LhqdU6ZQSl3hrA6TaqQ56PbAG/VOaK0XXZZk63BBae4I1fzUeFASP 0hbw== X-Gm-Message-State: AOAM530AWsPZyJ0KJM+yi5I4GgAmEPIaOQDUCAFVoOzZtAW773va1xUL 7LHTEr4um+08DBbQq7IVKKk= X-Google-Smtp-Source: ABdhPJzWKTfZxuv5pAX/GcihOwSizk30yBJX2yvw+9fTHGiHVFYYt0ZA3fiH8hp6oz2qZENOm8NSUw== X-Received: by 2002:a19:7d2:: with SMTP id 201mr2003647lfh.518.1611070637880; Tue, 19 Jan 2021 07:37:17 -0800 (PST) Received: from btopel-mobl.ger.intel.com (c213-102-90-208.bredband.comhem.se. [213.102.90.208]) by smtp.gmail.com with ESMTPSA id z2sm2309075lfd.142.2021.01.19.07.37.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 19 Jan 2021 07:37:17 -0800 (PST) From: =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= To: ast@kernel.org, daniel@iogearbox.net, netdev@vger.kernel.org, bpf@vger.kernel.org Cc: =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= , magnus.karlsson@intel.com, maciej.fijalkowski@intel.com, kuba@kernel.org, jonathan.lemon@gmail.com, maximmi@nvidia.com, davem@davemloft.net, hawk@kernel.org, john.fastabend@gmail.com, ciara.loftus@intel.com, weqaar.a.janjua@intel.com Subject: [PATCH bpf-next 7/8] selftest/bpf: add XDP socket tests for bpf_redirect_{xsk, map}() Date: Tue, 19 Jan 2021 16:36:54 +0100 Message-Id: <20210119153655.153999-8-bjorn.topel@gmail.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20210119153655.153999-1-bjorn.topel@gmail.com> References: <20210119153655.153999-1-bjorn.topel@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Björn Töpel Add support for externally loaded XDP programs to xdpxceiver/test_xsk.sh, so that bpf_redirect_xsk() and bpf_redirect_map() can be exercised. Signed-off-by: Björn Töpel --- tools/testing/selftests/bpf/test_xsk.sh | 48 +++++++++++++++ tools/testing/selftests/bpf/xdpxceiver.c | 77 ++++++++++++++++++++++-- tools/testing/selftests/bpf/xdpxceiver.h | 2 + 3 files changed, 123 insertions(+), 4 deletions(-) diff --git a/tools/testing/selftests/bpf/test_xsk.sh b/tools/testing/selftests/bpf/test_xsk.sh index 88a7483eaae4..3a3996edf527 100755 --- a/tools/testing/selftests/bpf/test_xsk.sh +++ b/tools/testing/selftests/bpf/test_xsk.sh @@ -245,6 +245,54 @@ retval=$? test_status $retval "${TEST_NAME}" statusList+=($retval) +### TEST 10 +TEST_NAME="SKB EXT BPF_REDIRECT_MAP" + +vethXDPgeneric ${VETH0} ${VETH1} ${NS1} + +params=("-S" "--ext-prog1") +execxdpxceiver params + +retval=$? +test_status $retval "${TEST_NAME}" +statusList+=($retval) + +### TEST 11 +TEST_NAME="DRV EXT BPF_REDIRECT_MAP" + +vethXDPnative ${VETH0} ${VETH1} ${NS1} + +params=("-N" "--ext-prog1") +execxdpxceiver params + +retval=$? +test_status $retval "${TEST_NAME}" +statusList+=($retval) + +### TEST 12 +TEST_NAME="SKB EXT BPF_REDIRECT_XSK" + +vethXDPgeneric ${VETH0} ${VETH1} ${NS1} + +params=("-S" "--ext-prog2") +execxdpxceiver params + +retval=$? +test_status $retval "${TEST_NAME}" +statusList+=($retval) + +### TEST 13 +TEST_NAME="DRV EXT BPF_REDIRECT_XSK" + +vethXDPnative ${VETH0} ${VETH1} ${NS1} + +params=("-N" "--ext-prog2") +execxdpxceiver params + +retval=$? +test_status $retval "${TEST_NAME}" +statusList+=($retval) + ## END TESTS cleanup_exit ${VETH0} ${VETH1} ${NS1} diff --git a/tools/testing/selftests/bpf/xdpxceiver.c b/tools/testing/selftests/bpf/xdpxceiver.c index 1e722ee76b1f..b88be91a6875 100644 --- a/tools/testing/selftests/bpf/xdpxceiver.c +++ b/tools/testing/selftests/bpf/xdpxceiver.c @@ -45,7 +45,7 @@ * - Only copy mode is supported because veth does not currently support * zero-copy mode * - * Total tests: 8 + * Total tests: 13 * * Flow: * ----- @@ -93,6 +93,7 @@ typedef __u16 __sum16; #include #include #include +#include #include "xdpxceiver.h" #include "../kselftest.h" @@ -296,6 +297,23 @@ static void xsk_populate_fill_ring(struct xsk_umem_info *umem) xsk_ring_prod__submit(&umem->fq, XSK_RING_PROD__DEFAULT_NUM_DESCS); } +static int update_xskmap(struct bpf_object *obj, struct xsk_socket_info *xsk) +{ + int xskmap, fd, key = opt_queue; + struct bpf_map *map; + + map = bpf_object__find_map_by_name(obj, "xsks_map"); + xskmap = bpf_map__fd(map); + if (xskmap < 0) + return 0; + + fd = xsk_socket__fd(xsk->xsk); + if (bpf_map_update_elem(xskmap, &key, &fd, 0)) + return -1; + + return 0; +} + static int xsk_configure_socket(struct ifobject *ifobject) { struct xsk_socket_config cfg; @@ -310,7 +328,7 @@ static int xsk_configure_socket(struct ifobject *ifobject) ifobject->xsk->umem = ifobject->umem; cfg.rx_size = XSK_RING_CONS__DEFAULT_NUM_DESCS; cfg.tx_size = XSK_RING_PROD__DEFAULT_NUM_DESCS; - cfg.libbpf_flags = 0; + cfg.libbpf_flags = ifobject->obj ? XSK_LIBBPF_FLAGS__INHIBIT_PROG_LOAD : 0; cfg.xdp_flags = opt_xdp_flags; cfg.bind_flags = opt_xdp_bind_flags; @@ -328,6 +346,11 @@ static int xsk_configure_socket(struct ifobject *ifobject) if (ret) return 1; + if (ifobject->obj) { + if (update_xskmap(ifobject->obj, ifobject->xsk)) + exit_with_error(errno); + } + return 0; } @@ -342,6 +365,8 @@ static struct option long_options[] = { {"bidi", optional_argument, 0, 'B'}, {"debug", optional_argument, 0, 'D'}, {"tx-pkt-count", optional_argument, 0, 'C'}, + {"ext-prog1", no_argument, 0, 1}, + {"ext-prog2", no_argument, 0, 1}, {0, 0, 0, 0} }; @@ -441,9 +466,30 @@ static int validate_interfaces(void) return ret; } +static int load_xdp_program(char *argv0, struct bpf_object **obj, int ext_prog) +{ + struct bpf_prog_load_attr prog_load_attr = { + .prog_type = BPF_PROG_TYPE_XDP, + }; + char xdp_filename[256]; + int prog_fd; + + snprintf(xdp_filename, sizeof(xdp_filename), "%s_ext%d.o", argv0, ext_prog); + prog_load_attr.file = xdp_filename; + + if (bpf_prog_load_xattr(&prog_load_attr, obj, &prog_fd)) + return -1; + return prog_fd; +} + +static int attach_xdp_program(int ifindex, int prog_fd) +{ + return bpf_set_link_xdp_fd(ifindex, prog_fd, opt_xdp_flags); +} + static void parse_command_line(int argc, char **argv) { - int option_index, interface_index = 0, c; + int option_index = 0, interface_index = 0, ext_prog = 0, c; opterr = 0; @@ -454,6 +500,9 @@ static void parse_command_line(int argc, char **argv) break; switch (c) { + case 1: + ext_prog = atoi(long_options[option_index].name + strlen("ext-prog")); + break; case 'i': if (interface_index == MAX_INTERFACES) break; @@ -509,6 +558,22 @@ static void parse_command_line(int argc, char **argv) usage(basename(argv[0])); ksft_exit_xfail(); } + + if (ext_prog) { + struct bpf_object *obj; + int prog_fd; + + for (int i = 0; i < MAX_INTERFACES; i++) { + prog_fd = load_xdp_program(argv[0], &obj, ext_prog); + if (prog_fd < 0) { + ksft_test_result_fail("ERROR: could not load external XDP program\n"); + ksft_exit_xfail(); + } + + ifdict[i]->prog_fd = prog_fd; + ifdict[i]->obj = obj; + } + } } static void kick_tx(struct xsk_socket_info *xsk) @@ -818,6 +883,7 @@ static void *worker_testapp_validate(void *arg) struct generic_data *data = (struct generic_data *)malloc(sizeof(struct generic_data)); struct iphdr *ip_hdr = (struct iphdr *)(pkt_data + sizeof(struct ethhdr)); struct ethhdr *eth_hdr = (struct ethhdr *)pkt_data; + struct ifobject *ifobject = (struct ifobject *)arg; void *bufs = NULL; pthread_attr_setstacksize(&attr, THREAD_STACK); @@ -830,6 +896,9 @@ static void *worker_testapp_validate(void *arg) if (strcmp(((struct ifobject *)arg)->nsname, "")) switch_namespace(((struct ifobject *)arg)->ifdict_index); + + if (ifobject->obj && attach_xdp_program(ifobject->ifindex, ifobject->prog_fd) < 0) + exit_with_error(errno); } if (((struct ifobject *)arg)->fv.vector == tx) { @@ -1035,7 +1104,7 @@ int main(int argc, char **argv) ifaceconfig->src_port = UDP_SRC_PORT; for (int i = 0; i < MAX_INTERFACES; i++) { - ifdict[i] = (struct ifobject *)malloc(sizeof(struct ifobject)); + ifdict[i] = (struct ifobject *)calloc(1, sizeof(struct ifobject)); if (!ifdict[i]) exit_with_error(errno); diff --git a/tools/testing/selftests/bpf/xdpxceiver.h b/tools/testing/selftests/bpf/xdpxceiver.h index 61f595b6f200..3c15c2e95026 100644 --- a/tools/testing/selftests/bpf/xdpxceiver.h +++ b/tools/testing/selftests/bpf/xdpxceiver.h @@ -124,6 +124,8 @@ struct ifobject { u32 src_ip; u16 src_port; u16 dst_port; + int prog_fd; + struct bpf_object *obj; }; static struct ifobject *ifdict[MAX_INTERFACES];