| Message ID | 20201120130026.19029-1-weqaar.a.janjua@intel.com |
|---|---|
| Headers | show |
| Series | selftests/bpf: xsk selftests | expand |
On 11/20/20 5:00 AM, Weqaar Janjua wrote: > This patch set adds AF_XDP selftests based on veth to selftests/bpf. > > # Topology: > # --------- > # ----------- > # _ | Process | _ > # / ----------- \ > # / | \ > # / | \ > # ----------- | ----------- > # | Thread1 | | | Thread2 | > # ----------- | ----------- > # | | | > # ----------- | ----------- > # | xskX | | | xskY | > # ----------- | ----------- > # | | | > # ----------- | ---------- > # | vethX | --------- | vethY | > # ----------- peer ---------- > # | | | > # namespaceX | namespaceY > > These selftests test AF_XDP SKB and Native/DRV modes using veth Virtual > Ethernet interfaces. > > The test program contains two threads, each thread is single socket with > a unique UMEM. It validates in-order packet delivery and packet content > by sending packets to each other. > > Prerequisites setup by script test_xsk_prerequisites.sh: > > Set up veth interfaces as per the topology shown ^^: > * setup two veth interfaces and one namespace > ** veth<xxxx> in root namespace > ** veth<yyyy> in af_xdp<xxxx> namespace > ** namespace af_xdp<xxxx> > * create a spec file veth.spec that includes this run-time configuration > that is read by test scripts - filenames prefixed with test_xsk_ > *** xxxx and yyyy are randomly generated 4 digit numbers used to avoid > conflict with any existing interface > > The following tests are provided: > > 1. AF_XDP SKB mode > Generic mode XDP is driver independent, used when the driver does > not have support for XDP. Works on any netdevice using sockets and > generic XDP path. XDP hook from netif_receive_skb(). > a. nopoll - soft-irq processing > b. poll - using poll() syscall > c. Socket Teardown > Create a Tx and a Rx socket, Tx from one socket, Rx on another. > Destroy both sockets, then repeat multiple times. Only nopoll mode > is used > d. Bi-directional Sockets > Configure sockets as bi-directional tx/rx sockets, sets up fill > and completion rings on each socket, tx/rx in both directions. > Only nopoll mode is used > > 2. AF_XDP DRV/Native mode > Works on any netdevice with XDP_REDIRECT support, driver dependent. > Processes packets before SKB allocation. Provides better performance > than SKB. Driver hook available just after DMA of buffer descriptor. > a. nopoll > b. poll > c. Socket Teardown > d. Bi-directional Sockets > * Only copy mode is supported because veth does not currently support > zero-copy mode > > Total tests: 8 > > Flow: > * Single process spawns two threads: Tx and Rx > * Each of these two threads attach to a veth interface within their > assigned namespaces > * Each thread creates one AF_XDP socket connected to a unique umem > for each veth interface > * Tx thread transmits 10k packets from veth<xxxx> to veth<yyyy> > * Rx thread verifies if all 10k packets were received and delivered > in-order, and have the right content > > v2 changes: > * Move selftests/xsk to selftests/bpf > * Remove Makefiles under selftests/xsk, and utilize selftests/bpf/Makefile > > Structure of the patch set: > > Patch 1: This patch adds XSK Selftests framework under selftests/bpf > Patch 2: Adds tests: SKB poll and nopoll mode, and mac-ip-udp debug > Patch 3: Adds tests: DRV poll and nopoll mode > Patch 4: Adds tests: SKB and DRV Socket Teardown > Patch 5: Adds tests: SKB and DRV Bi-directional Sockets I just want to report that after applying the above 5 patches on top of bpf-next commit 450d060e8f75 ("bpftool: Add {i,d}tlb_misses support for bpftool profile"), I hit the following error with below command sequences: $ ./test_xsk_prerequisites.sh $ ./test_xsk_skb_poll.sh # Interface found: ve1480 # Interface found: ve9258 # NS switched: af_xdp9258 1..1 # Interface [ve9258] vector [Rx] # Interface [ve1480] vector [Tx] # Sending 10000 packets on interface ve1480 [ 331.741244] ------------[ cut here ]------------ [ 331.741741] kernel BUG at net/core/skbuff.c:1621! [ 331.742265] invalid opcode: 0000 [#1] PREEMPT SMP PTI [ 331.742837] CPU: 0 PID: 1883 Comm: xdpxceiver Not tainted 5.10.0-rc3+ #1037 [ 331.743468] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.9.3 -1.el7.centos 04/01/2014 [ 331.744300] RIP: 0010:pskb_expand_head+0x27b/0x310 [ 331.744747] Code: df e8 69 fc ff ff e9 ab fe ff ff 44 2b 6c 24 04 44 01 ab d0 00 00 00 48 83 c4 08 31 c0 5b 5d 41 5c 41 5d 41 5e 41 5f c3 0f 0b <0f> 0b be 02 00 00 00 e8 89 4e ad ff 8b 83 bc 00 00 00 48 03 83 c0 [ 331.746414] RSP: 0018:ffffbae4c0003d08 EFLAGS: 00010202 [ 331.746892] RAX: 000000000000013f RBX: ffff9e0a8367ad00 RCX: 0000000000000a20 [ 331.747534] RDX: 0000000000000002 RSI: 0000000000000100 RDI: ffff9e0a8367ad00 [ 331.748192] RBP: ffffbae4c00b2000 R08: 0000000000000001 R09: 000000000000000e [ 331.748834] R10: ffffbae4c0003eb8 R11: 00000000ef974e19 R12: ffff9e0a86ecf000 [ 331.749472] R13: 0000000000000001 R14: ffff9e0a8367ad00 R15: ffff9e0a8367ad00 [ 331.750119] FS: 00007ff0806c5e00(0000) GS:ffff9e0abae00000(0000) knlGS:00000 00000000000 [ 331.750848] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 331.751379] CR2: 00007ff0806c01d8 CR3: 0000000106e00006 CR4: 0000000000370ef0 [ 331.752022] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 331.752665] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 331.753307] Call Trace: [ 331.753535] <IRQ> [ 331.753736] do_xdp_generic.part.157+0xa3/0x550 [ 331.754151] __netif_receive_skb_core+0x67e/0x12b0 [ 331.754588] ? process_backlog+0x86/0x250 [ 331.754961] ? __netif_receive_skb_one_core+0x3c/0xa0 [ 331.755419] __netif_receive_skb_one_core+0x3c/0xa0 [ 331.755865] process_backlog+0xf5/0x250 [ 331.756215] net_rx_action+0x144/0x440 [ 331.756559] __do_softirq+0xe4/0x493 [ 331.756894] asm_call_irq_on_stack+0x12/0x20 [ 331.757282] </IRQ> [ 331.757478] ? dev_direct_xmit+0x1e8/0x230 [ 331.757856] do_softirq_own_stack+0x81/0xa0 [ 331.758244] do_softirq.part.16+0x3c/0x80 [ 331.758611] __local_bh_enable_ip+0xda/0xe0 [ 331.758995] dev_direct_xmit+0x20d/0x230 [ 331.759356] __xsk_sendmsg+0x314/0x3d0 [ 331.759704] sock_sendmsg+0x5b/0x60 [ 331.760025] __sys_sendto+0xf1/0x160 [ 331.760355] ? lockdep_hardirqs_on+0xbf/0x130 [ 331.760759] ? syscall_enter_from_user_mode+0x1c/0x50 [ 331.761216] __x64_sys_sendto+0x24/0x30 [ 331.761563] do_syscall_64+0x33/0x40 [ 331.761895] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [ 331.762357] RIP: 0033:0x7ff08c8b9633 [ 331.762689] Code: 49 89 ca b8 2c 00 00 00 0f 05 48 3d 01 f0 ff ff 73 34 c3 48 83 ec 08 e8 1b f7 ff ff 48 89 04 24 49 89 ca b8 2c 00 00 00 0f 05 <48> 8b 3c 24 48 89 c2 e8 61 f7 ff ff 48 89 d0 48 83 c4 08 48 3d 01 [ 331.764356] RSP: 002b:00007ff0806c55c0 EFLAGS: 00000293 ORIG_RAX: 00000000000 0002c [ 331.765038] RAX: ffffffffffffffda RBX: 00007ff0780009b0 RCX: 00007ff08c8b9633 [ 331.765684] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000008 [ 331.766324] RBP: 0000000000000040 R08: 0000000000000000 R09: 0000000000000000 [ 331.766969] R10: 0000000000000040 R11: 0000000000000293 R12: 0000000000000040 [ 331.767608] R13: 0000000000000040 R14: 0000000000000000 R15: 0000000000cd4030 [ 331.768261] Modules linked in: [ 331.768596] ---[ end trace d9ca37a7957928dd ]--- [ 331.769126] RIP: 0010:pskb_expand_head+0x27b/0x310 [ 331.769678] Code: df e8 69 fc ff ff e9 ab fe ff ff 44 2b 6c 24 04 44 01 ab d0 00 00 00 48 83 c4 08 31 c0 5b 5d 41 5c 41 5d 41 5e 41 5f c3 0f 0b <0f> 0b be 02 00 00 00 e8 89 4e ad ff 8b 83 bc 00 00 00 48 03 83 c0 [ 331.771459] RSP: 0018:ffffbae4c0003d08 EFLAGS: 00010202 [ 331.772043] RAX: 000000000000013f RBX: ffff9e0a8367ad00 RCX: 0000000000000a20 [ 331.772784] RDX: 0000000000000002 RSI: 0000000000000100 RDI: ffff9e0a8367ad00 [ 331.773526] RBP: ffffbae4c00b2000 R08: 0000000000000001 R09: 000000000000000e [ 331.774293] R10: ffffbae4c0003eb8 R11: 00000000ef974e19 R12: ffff9e0a86ecf000 [ 331.775049] R13: 0000000000000001 R14: ffff9e0a8367ad00 R15: ffff9e0a8367ad00 [ 331.775901] FS: 00007ff0806c5e00(0000) GS:ffff9e0abae00000(0000) knlGS:00000 00000000000 [ 331.776809] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 331.777455] CR2: 00007ff0806c01d8 CR3: 0000000106e00006 CR4: 0000000000370ef0 [ 331.778232] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 331.778989] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 331.779739] Kernel panic - not syncing: Fatal exception in interrupt [ 331.780523] Kernel Offset: 0xb800000 from 0xffffffff81000000 (relocation rang e: 0xffffffff80000000-0xffffffffbfffffff) [ 331.781488] ---[ end Kernel panic - not syncing: Fatal exception in interrupt ]--- In any case, kernel should not panic. You or somebody familiar with xsk may want to take a look. > > Thanks: Weqaar > > Weqaar Janjua (5): > selftests/bpf: xsk selftests framework > selftests/bpf: xsk selftests - SKB POLL, NOPOLL > selftests/bpf: xsk selftests - DRV POLL, NOPOLL > selftests/bpf: xsk selftests - Socket Teardown - SKB, DRV > selftests/bpf: xsk selftests - Bi-directional Sockets - SKB, DRV > > tools/testing/selftests/bpf/Makefile | 15 +- > .../bpf/test_xsk_drv_bidirectional.sh | 23 + > .../selftests/bpf/test_xsk_drv_nopoll.sh | 20 + > .../selftests/bpf/test_xsk_drv_poll.sh | 20 + > .../selftests/bpf/test_xsk_drv_teardown.sh | 20 + > .../selftests/bpf/test_xsk_prerequisites.sh | 127 ++ > .../bpf/test_xsk_skb_bidirectional.sh | 20 + > .../selftests/bpf/test_xsk_skb_nopoll.sh | 20 + > .../selftests/bpf/test_xsk_skb_poll.sh | 20 + > .../selftests/bpf/test_xsk_skb_teardown.sh | 20 + > tools/testing/selftests/bpf/xdpxceiver.c | 1056 +++++++++++++++++ > tools/testing/selftests/bpf/xdpxceiver.h | 158 +++ > tools/testing/selftests/bpf/xsk_env.sh | 28 + > tools/testing/selftests/bpf/xsk_prereqs.sh | 119 ++ > 14 files changed, 1664 insertions(+), 2 deletions(-) [...]
On Fri, 20 Nov 2020 at 20:45, Yonghong Song <yhs@fb.com> wrote: > > > > On 11/20/20 5:00 AM, Weqaar Janjua wrote: > > Adds following tests: > > > > 1. AF_XDP SKB mode > > d. Bi-directional Sockets > > Configure sockets as bi-directional tx/rx sockets, sets up fill > > and completion rings on each socket, tx/rx in both directions. > > Only nopoll mode is used > > > > 2. AF_XDP DRV/Native mode > > d. Bi-directional Sockets > > * Only copy mode is supported because veth does not currently support > > zero-copy mode > > > > Signed-off-by: Weqaar Janjua <weqaar.a.janjua@intel.com> > > --- > > tools/testing/selftests/bpf/Makefile | 4 +- > > .../bpf/test_xsk_drv_bidirectional.sh | 23 ++++ > > .../selftests/bpf/test_xsk_drv_teardown.sh | 3 - > > .../bpf/test_xsk_skb_bidirectional.sh | 20 ++++ > > tools/testing/selftests/bpf/xdpxceiver.c | 100 +++++++++++++----- > > tools/testing/selftests/bpf/xdpxceiver.h | 4 + > > 6 files changed, 126 insertions(+), 28 deletions(-) > > create mode 100755 tools/testing/selftests/bpf/test_xsk_drv_bidirectional.sh > > create mode 100755 tools/testing/selftests/bpf/test_xsk_skb_bidirectional.sh > > > > diff --git a/tools/testing/selftests/bpf/Makefile b/tools/testing/selftests/bpf/Makefile > > index 515b29d321d7..258bd72812e0 100644 > > --- a/tools/testing/selftests/bpf/Makefile > > +++ b/tools/testing/selftests/bpf/Makefile > > @@ -78,7 +78,9 @@ TEST_PROGS := test_kmod.sh \ > > test_xsk_drv_nopoll.sh \ > > test_xsk_drv_poll.sh \ > > test_xsk_skb_teardown.sh \ > > - test_xsk_drv_teardown.sh > > + test_xsk_drv_teardown.sh \ > > + test_xsk_skb_bidirectional.sh \ > > + test_xsk_drv_bidirectional.sh > > > > TEST_PROGS_EXTENDED := with_addr.sh \ > > with_tunnels.sh \ > > diff --git a/tools/testing/selftests/bpf/test_xsk_drv_bidirectional.sh b/tools/testing/selftests/bpf/test_xsk_drv_bidirectional.sh > > new file mode 100755 > > index 000000000000..d3a7e2934d83 > > --- /dev/null > > +++ b/tools/testing/selftests/bpf/test_xsk_drv_bidirectional.sh > > @@ -0,0 +1,23 @@ > > +#!/bin/bash > > +# SPDX-License-Identifier: GPL-2.0 > > +# Copyright(c) 2020 Intel Corporation. > > + > > +# See test_xsk_prerequisites.sh for detailed information on tests > > + > > +. xsk_prereqs.sh > > +. xsk_env.sh > > + > > +TEST_NAME="DRV BIDIRECTIONAL SOCKETS" > > + > > +vethXDPnative ${VETH0} ${VETH1} ${NS1} > > + > > +params=("-N" "-B") > > +execxdpxceiver params > > + > > +retval=$? > > +test_status $retval "${TEST_NAME}" > > + > > +# Must be called in the last test to execute > > +cleanup_exit ${VETH0} ${VETH1} ${NS1} > > This also makes hard to run tests as users will not know this unless > they are familiar with the details of the tests. > > How about you have another scripts test_xsk.sh which includes all these > individual tests and pull the above cleanup_exit into test_xsk.sh? > User just need to run test_xsk.sh will be able to run all tests you > implemented here. > This works, test_xsk_* >> test_xsk.sh, will ship out as v3. > > + > > +test_exit $retval 0 > > diff --git a/tools/testing/selftests/bpf/test_xsk_drv_teardown.sh b/tools/testing/selftests/bpf/test_xsk_drv_teardown.sh > [...]
On 2020-11-21 01:31, Yonghong Song wrote: > > > On 11/20/20 5:00 AM, Weqaar Janjua wrote: >> This patch set adds AF_XDP selftests based on veth to selftests/bpf. >> >> # Topology: >> # --------- >> # ----------- >> # _ | Process | _ >> # / ----------- \ >> # / | \ >> # / | \ >> # ----------- | ----------- >> # | Thread1 | | | Thread2 | >> # ----------- | ----------- >> # | | | >> # ----------- | ----------- >> # | xskX | | | xskY | >> # ----------- | ----------- >> # | | | >> # ----------- | ---------- >> # | vethX | --------- | vethY | >> # ----------- peer ---------- >> # | | | >> # namespaceX | namespaceY >> >> These selftests test AF_XDP SKB and Native/DRV modes using veth Virtual >> Ethernet interfaces. >> >> The test program contains two threads, each thread is single socket with >> a unique UMEM. It validates in-order packet delivery and packet content >> by sending packets to each other. >> >> Prerequisites setup by script test_xsk_prerequisites.sh: >> >> Set up veth interfaces as per the topology shown ^^: >> * setup two veth interfaces and one namespace >> ** veth<xxxx> in root namespace >> ** veth<yyyy> in af_xdp<xxxx> namespace >> ** namespace af_xdp<xxxx> >> * create a spec file veth.spec that includes this run-time >> configuration >> that is read by test scripts - filenames prefixed with test_xsk_ >> *** xxxx and yyyy are randomly generated 4 digit numbers used to >> avoid >> conflict with any existing interface >> >> The following tests are provided: >> >> 1. AF_XDP SKB mode >> Generic mode XDP is driver independent, used when the driver does >> not have support for XDP. Works on any netdevice using sockets and >> generic XDP path. XDP hook from netif_receive_skb(). >> a. nopoll - soft-irq processing >> b. poll - using poll() syscall >> c. Socket Teardown >> Create a Tx and a Rx socket, Tx from one socket, Rx on another. >> Destroy both sockets, then repeat multiple times. Only nopoll mode >> is used >> d. Bi-directional Sockets >> Configure sockets as bi-directional tx/rx sockets, sets up fill >> and completion rings on each socket, tx/rx in both directions. >> Only nopoll mode is used >> >> 2. AF_XDP DRV/Native mode >> Works on any netdevice with XDP_REDIRECT support, driver dependent. >> Processes packets before SKB allocation. Provides better performance >> than SKB. Driver hook available just after DMA of buffer descriptor. >> a. nopoll >> b. poll >> c. Socket Teardown >> d. Bi-directional Sockets >> * Only copy mode is supported because veth does not currently support >> zero-copy mode >> >> Total tests: 8 >> >> Flow: >> * Single process spawns two threads: Tx and Rx >> * Each of these two threads attach to a veth interface within their >> assigned namespaces >> * Each thread creates one AF_XDP socket connected to a unique umem >> for each veth interface >> * Tx thread transmits 10k packets from veth<xxxx> to veth<yyyy> >> * Rx thread verifies if all 10k packets were received and delivered >> in-order, and have the right content >> >> v2 changes: >> * Move selftests/xsk to selftests/bpf >> * Remove Makefiles under selftests/xsk, and utilize >> selftests/bpf/Makefile >> >> Structure of the patch set: >> >> Patch 1: This patch adds XSK Selftests framework under selftests/bpf >> Patch 2: Adds tests: SKB poll and nopoll mode, and mac-ip-udp debug >> Patch 3: Adds tests: DRV poll and nopoll mode >> Patch 4: Adds tests: SKB and DRV Socket Teardown >> Patch 5: Adds tests: SKB and DRV Bi-directional Sockets > > I just want to report that after applying the above 5 patches > on top of bpf-next commit 450d060e8f75 ("bpftool: Add {i,d}tlb_misses > support for bpftool profile"), I hit the following error with below > command sequences: > > $ ./test_xsk_prerequisites.sh > $ ./test_xsk_skb_poll.sh > # Interface found: ve1480 > # Interface found: ve9258 > # NS switched: af_xdp9258 > 1..1 > # Interface [ve9258] vector [Rx] > # Interface [ve1480] vector [Tx] > # Sending 10000 packets on interface ve1480 > [ 331.741244] ------------[ cut here ]------------ > [ 331.741741] kernel BUG at net/core/skbuff.c:1621! > [ 331.742265] invalid opcode: 0000 [#1] PREEMPT SMP PTI > [ 331.742837] CPU: 0 PID: 1883 Comm: xdpxceiver Not tainted 5.10.0-rc3+ > #1037 > [ 331.743468] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), > BIOS 1.9.3 > -1.el7.centos 04/01/2014 > [ 331.744300] RIP: 0010:pskb_expand_head+0x27b/0x310 Ugh, looks like the tests are working. :-P This is a BUG_ON(skb_shared(skb)) trigger, related to the skbuff refcount changes done recently in AF_XDP. I'll cook a patch! Thanks for the report! Björn
On 2020-11-23 13:20, Björn Töpel wrote: > On 2020-11-21 01:31, Yonghong Song wrote: >> >> >> On 11/20/20 5:00 AM, Weqaar Janjua wrote: >>> This patch set adds AF_XDP selftests based on veth to selftests/bpf. >>> >>> # Topology: >>> # --------- >>> # ----------- >>> # _ | Process | _ >>> # / ----------- \ >>> # / | \ >>> # / | \ >>> # ----------- | ----------- >>> # | Thread1 | | | Thread2 | >>> # ----------- | ----------- >>> # | | | >>> # ----------- | ----------- >>> # | xskX | | | xskY | >>> # ----------- | ----------- >>> # | | | >>> # ----------- | ---------- >>> # | vethX | --------- | vethY | >>> # ----------- peer ---------- >>> # | | | >>> # namespaceX | namespaceY >>> >>> These selftests test AF_XDP SKB and Native/DRV modes using veth Virtual >>> Ethernet interfaces. >>> >>> The test program contains two threads, each thread is single socket with >>> a unique UMEM. It validates in-order packet delivery and packet content >>> by sending packets to each other. >>> >>> Prerequisites setup by script test_xsk_prerequisites.sh: >>> >>> Set up veth interfaces as per the topology shown ^^: >>> * setup two veth interfaces and one namespace >>> ** veth<xxxx> in root namespace >>> ** veth<yyyy> in af_xdp<xxxx> namespace >>> ** namespace af_xdp<xxxx> >>> * create a spec file veth.spec that includes this run-time >>> configuration >>> that is read by test scripts - filenames prefixed with test_xsk_ >>> *** xxxx and yyyy are randomly generated 4 digit numbers used to >>> avoid >>> conflict with any existing interface >>> >>> The following tests are provided: >>> >>> 1. AF_XDP SKB mode >>> Generic mode XDP is driver independent, used when the driver does >>> not have support for XDP. Works on any netdevice using sockets and >>> generic XDP path. XDP hook from netif_receive_skb(). >>> a. nopoll - soft-irq processing >>> b. poll - using poll() syscall >>> c. Socket Teardown >>> Create a Tx and a Rx socket, Tx from one socket, Rx on another. >>> Destroy both sockets, then repeat multiple times. Only nopoll >>> mode >>> is used >>> d. Bi-directional Sockets >>> Configure sockets as bi-directional tx/rx sockets, sets up fill >>> and completion rings on each socket, tx/rx in both directions. >>> Only nopoll mode is used >>> >>> 2. AF_XDP DRV/Native mode >>> Works on any netdevice with XDP_REDIRECT support, driver dependent. >>> Processes packets before SKB allocation. Provides better performance >>> than SKB. Driver hook available just after DMA of buffer descriptor. >>> a. nopoll >>> b. poll >>> c. Socket Teardown >>> d. Bi-directional Sockets >>> * Only copy mode is supported because veth does not currently >>> support >>> zero-copy mode >>> >>> Total tests: 8 >>> >>> Flow: >>> * Single process spawns two threads: Tx and Rx >>> * Each of these two threads attach to a veth interface within their >>> assigned namespaces >>> * Each thread creates one AF_XDP socket connected to a unique umem >>> for each veth interface >>> * Tx thread transmits 10k packets from veth<xxxx> to veth<yyyy> >>> * Rx thread verifies if all 10k packets were received and delivered >>> in-order, and have the right content >>> >>> v2 changes: >>> * Move selftests/xsk to selftests/bpf >>> * Remove Makefiles under selftests/xsk, and utilize >>> selftests/bpf/Makefile >>> >>> Structure of the patch set: >>> >>> Patch 1: This patch adds XSK Selftests framework under selftests/bpf >>> Patch 2: Adds tests: SKB poll and nopoll mode, and mac-ip-udp debug >>> Patch 3: Adds tests: DRV poll and nopoll mode >>> Patch 4: Adds tests: SKB and DRV Socket Teardown >>> Patch 5: Adds tests: SKB and DRV Bi-directional Sockets >> >> I just want to report that after applying the above 5 patches >> on top of bpf-next commit 450d060e8f75 ("bpftool: Add {i,d}tlb_misses >> support for bpftool profile"), I hit the following error with below >> command sequences: >> >> $ ./test_xsk_prerequisites.sh >> $ ./test_xsk_skb_poll.sh >> # Interface found: ve1480 >> # Interface found: ve9258 >> # NS switched: af_xdp9258 >> 1..1 >> # Interface [ve9258] vector [Rx] >> # Interface [ve1480] vector [Tx] >> # Sending 10000 packets on interface ve1480 >> [ 331.741244] ------------[ cut here ]------------ >> [ 331.741741] kernel BUG at net/core/skbuff.c:1621! >> [ 331.742265] invalid opcode: 0000 [#1] PREEMPT SMP PTI >> [ 331.742837] CPU: 0 PID: 1883 Comm: xdpxceiver Not tainted >> 5.10.0-rc3+ #1037 >> [ 331.743468] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), >> BIOS 1.9.3 >> -1.el7.centos 04/01/2014 >> [ 331.744300] RIP: 0010:pskb_expand_head+0x27b/0x310 > > Ugh, looks like the tests are working. :-P > > This is a BUG_ON(skb_shared(skb)) trigger, related to the skbuff > refcount changes done recently in AF_XDP. > > I'll cook a patch! Thanks for the report! > Posted a fix [1]. Please not that it's for the bpf tree, so when Weqaar pushes the v3 of the selftests to bpf-next, [1] needs to be pulled in. Björn [1] https://lore.kernel.org/bpf/20201123131215.136131-1-bjorn.topel@gmail.com/ > > Björn
On 11/24/20 7:11 AM, Weqaar Janjua wrote: > On Sat, 21 Nov 2020 at 20:14, Weqaar Janjua <weqaar.janjua@gmail.com> wrote: >> >> On Fri, 20 Nov 2020 at 20:45, Yonghong Song <yhs@fb.com> wrote: >>> >>> >>> >>> On 11/20/20 5:00 AM, Weqaar Janjua wrote: >>>> Adds following tests: >>>> >>>> 1. AF_XDP SKB mode >>>> d. Bi-directional Sockets >>>> Configure sockets as bi-directional tx/rx sockets, sets up fill >>>> and completion rings on each socket, tx/rx in both directions. >>>> Only nopoll mode is used >>>> >>>> 2. AF_XDP DRV/Native mode >>>> d. Bi-directional Sockets >>>> * Only copy mode is supported because veth does not currently support >>>> zero-copy mode >>>> >>>> Signed-off-by: Weqaar Janjua <weqaar.a.janjua@intel.com> >>>> --- >>>> tools/testing/selftests/bpf/Makefile | 4 +- >>>> .../bpf/test_xsk_drv_bidirectional.sh | 23 ++++ >>>> .../selftests/bpf/test_xsk_drv_teardown.sh | 3 - >>>> .../bpf/test_xsk_skb_bidirectional.sh | 20 ++++ >>>> tools/testing/selftests/bpf/xdpxceiver.c | 100 +++++++++++++----- >>>> tools/testing/selftests/bpf/xdpxceiver.h | 4 + >>>> 6 files changed, 126 insertions(+), 28 deletions(-) >>>> create mode 100755 tools/testing/selftests/bpf/test_xsk_drv_bidirectional.sh >>>> create mode 100755 tools/testing/selftests/bpf/test_xsk_skb_bidirectional.sh >>>> >>>> diff --git a/tools/testing/selftests/bpf/Makefile b/tools/testing/selftests/bpf/Makefile >>>> index 515b29d321d7..258bd72812e0 100644 >>>> --- a/tools/testing/selftests/bpf/Makefile >>>> +++ b/tools/testing/selftests/bpf/Makefile >>>> @@ -78,7 +78,9 @@ TEST_PROGS := test_kmod.sh \ >>>> test_xsk_drv_nopoll.sh \ >>>> test_xsk_drv_poll.sh \ >>>> test_xsk_skb_teardown.sh \ >>>> - test_xsk_drv_teardown.sh >>>> + test_xsk_drv_teardown.sh \ >>>> + test_xsk_skb_bidirectional.sh \ >>>> + test_xsk_drv_bidirectional.sh >>>> >>>> TEST_PROGS_EXTENDED := with_addr.sh \ >>>> with_tunnels.sh \ >>>> diff --git a/tools/testing/selftests/bpf/test_xsk_drv_bidirectional.sh b/tools/testing/selftests/bpf/test_xsk_drv_bidirectional.sh >>>> new file mode 100755 >>>> index 000000000000..d3a7e2934d83 >>>> --- /dev/null >>>> +++ b/tools/testing/selftests/bpf/test_xsk_drv_bidirectional.sh >>>> @@ -0,0 +1,23 @@ >>>> +#!/bin/bash >>>> +# SPDX-License-Identifier: GPL-2.0 >>>> +# Copyright(c) 2020 Intel Corporation. >>>> + >>>> +# See test_xsk_prerequisites.sh for detailed information on tests >>>> + >>>> +. xsk_prereqs.sh >>>> +. xsk_env.sh >>>> + >>>> +TEST_NAME="DRV BIDIRECTIONAL SOCKETS" >>>> + >>>> +vethXDPnative ${VETH0} ${VETH1} ${NS1} >>>> + >>>> +params=("-N" "-B") >>>> +execxdpxceiver params >>>> + >>>> +retval=$? >>>> +test_status $retval "${TEST_NAME}" >>>> + >>>> +# Must be called in the last test to execute >>>> +cleanup_exit ${VETH0} ${VETH1} ${NS1} >>> >>> This also makes hard to run tests as users will not know this unless >>> they are familiar with the details of the tests. >>> >>> How about you have another scripts test_xsk.sh which includes all these >>> individual tests and pull the above cleanup_exit into test_xsk.sh? >>> User just need to run test_xsk.sh will be able to run all tests you >>> implemented here. >>> >> This works, test_xsk_* >> test_xsk.sh, will ship out as v3. >> > An issue with merging all tests in a single test_xsk.sh is reporting > number of test failures, with this approach a single test status is > printed by kselftest: > > # PREREQUISITES: [ PASS ] > # SKB NOPOLL: [ FAIL ] > # SKB POLL: [ PASS ] > ok 1 selftests: xsk-patch2: test_xsk.sh > > This is due to the fact Makefile has one TEST_PROGS = test_xsk.sh > (thus kselftest considers it one test?), where in the original > approach all tests have separate TEST_PROGS .sh which makes reporting > match each test and status. This can be a problem for automation. > > An alternative would be to exit each test with failure status but then > the tests will stop execution at the failed test without executing the > rest of xsk tests, which we probably wouldn't want. > > Suggestions please? I think it is okay to put everything xsk related to one test. If later on the test becomes more complex, you can have test_xsk_<1>.sh test_xsk_<2>.sh etc. But each .sh should be able to run independently without any particular order. You can have subtests inside the .sh file. See test_offload.py as an example. You do not need to exit after one subtest fails, you can continue to run the next one. currently test_offload.py may exit when some subtest failed, but I think you don't have to. > >>>> + >>>> +test_exit $retval 0 >>>> diff --git a/tools/testing/selftests/bpf/test_xsk_drv_teardown.sh b/tools/testing/selftests/bpf/test_xsk_drv_teardown.sh >>> [...]
On Tue, 24 Nov 2020 at 17:10, Yonghong Song <yhs@fb.com> wrote: > > > > On 11/24/20 7:11 AM, Weqaar Janjua wrote: > > On Sat, 21 Nov 2020 at 20:14, Weqaar Janjua <weqaar.janjua@gmail.com> wrote: > >> > >> On Fri, 20 Nov 2020 at 20:45, Yonghong Song <yhs@fb.com> wrote: > >>> > >>> > >>> > >>> On 11/20/20 5:00 AM, Weqaar Janjua wrote: > >>>> Adds following tests: > >>>> > >>>> 1. AF_XDP SKB mode > >>>> d. Bi-directional Sockets > >>>> Configure sockets as bi-directional tx/rx sockets, sets up fill > >>>> and completion rings on each socket, tx/rx in both directions. > >>>> Only nopoll mode is used > >>>> > >>>> 2. AF_XDP DRV/Native mode > >>>> d. Bi-directional Sockets > >>>> * Only copy mode is supported because veth does not currently support > >>>> zero-copy mode > >>>> > >>>> Signed-off-by: Weqaar Janjua <weqaar.a.janjua@intel.com> > >>>> --- > >>>> tools/testing/selftests/bpf/Makefile | 4 +- > >>>> .../bpf/test_xsk_drv_bidirectional.sh | 23 ++++ > >>>> .../selftests/bpf/test_xsk_drv_teardown.sh | 3 - > >>>> .../bpf/test_xsk_skb_bidirectional.sh | 20 ++++ > >>>> tools/testing/selftests/bpf/xdpxceiver.c | 100 +++++++++++++----- > >>>> tools/testing/selftests/bpf/xdpxceiver.h | 4 + > >>>> 6 files changed, 126 insertions(+), 28 deletions(-) > >>>> create mode 100755 tools/testing/selftests/bpf/test_xsk_drv_bidirectional.sh > >>>> create mode 100755 tools/testing/selftests/bpf/test_xsk_skb_bidirectional.sh > >>>> > >>>> diff --git a/tools/testing/selftests/bpf/Makefile b/tools/testing/selftests/bpf/Makefile > >>>> index 515b29d321d7..258bd72812e0 100644 > >>>> --- a/tools/testing/selftests/bpf/Makefile > >>>> +++ b/tools/testing/selftests/bpf/Makefile > >>>> @@ -78,7 +78,9 @@ TEST_PROGS := test_kmod.sh \ > >>>> test_xsk_drv_nopoll.sh \ > >>>> test_xsk_drv_poll.sh \ > >>>> test_xsk_skb_teardown.sh \ > >>>> - test_xsk_drv_teardown.sh > >>>> + test_xsk_drv_teardown.sh \ > >>>> + test_xsk_skb_bidirectional.sh \ > >>>> + test_xsk_drv_bidirectional.sh > >>>> > >>>> TEST_PROGS_EXTENDED := with_addr.sh \ > >>>> with_tunnels.sh \ > >>>> diff --git a/tools/testing/selftests/bpf/test_xsk_drv_bidirectional.sh b/tools/testing/selftests/bpf/test_xsk_drv_bidirectional.sh > >>>> new file mode 100755 > >>>> index 000000000000..d3a7e2934d83 > >>>> --- /dev/null > >>>> +++ b/tools/testing/selftests/bpf/test_xsk_drv_bidirectional.sh > >>>> @@ -0,0 +1,23 @@ > >>>> +#!/bin/bash > >>>> +# SPDX-License-Identifier: GPL-2.0 > >>>> +# Copyright(c) 2020 Intel Corporation. > >>>> + > >>>> +# See test_xsk_prerequisites.sh for detailed information on tests > >>>> + > >>>> +. xsk_prereqs.sh > >>>> +. xsk_env.sh > >>>> + > >>>> +TEST_NAME="DRV BIDIRECTIONAL SOCKETS" > >>>> + > >>>> +vethXDPnative ${VETH0} ${VETH1} ${NS1} > >>>> + > >>>> +params=("-N" "-B") > >>>> +execxdpxceiver params > >>>> + > >>>> +retval=$? > >>>> +test_status $retval "${TEST_NAME}" > >>>> + > >>>> +# Must be called in the last test to execute > >>>> +cleanup_exit ${VETH0} ${VETH1} ${NS1} > >>> > >>> This also makes hard to run tests as users will not know this unless > >>> they are familiar with the details of the tests. > >>> > >>> How about you have another scripts test_xsk.sh which includes all these > >>> individual tests and pull the above cleanup_exit into test_xsk.sh? > >>> User just need to run test_xsk.sh will be able to run all tests you > >>> implemented here. > >>> > >> This works, test_xsk_* >> test_xsk.sh, will ship out as v3. > >> > > An issue with merging all tests in a single test_xsk.sh is reporting > > number of test failures, with this approach a single test status is > > printed by kselftest: > > > > # PREREQUISITES: [ PASS ] > > # SKB NOPOLL: [ FAIL ] > > # SKB POLL: [ PASS ] > > ok 1 selftests: xsk-patch2: test_xsk.sh > > > > This is due to the fact Makefile has one TEST_PROGS = test_xsk.sh > > (thus kselftest considers it one test?), where in the original > > approach all tests have separate TEST_PROGS .sh which makes reporting > > match each test and status. This can be a problem for automation. > > > > An alternative would be to exit each test with failure status but then > > the tests will stop execution at the failed test without executing the > > rest of xsk tests, which we probably wouldn't want. > > > > Suggestions please? > > I think it is okay to put everything xsk related to one test. > If later on the test becomes more complex, you can have > test_xsk_<1>.sh test_xsk_<2>.sh etc. But each .sh should be able to > run independently without any particular order. > > You can have subtests inside the .sh file. See test_offload.py as > an example. You do not need to exit after one subtest fails, you can > continue to run the next one. currently test_offload.py > may exit when some subtest failed, but I think you don't have to. > ACK, I will go ahead and merge all test_xsk_*.sh into test_xsk.sh. Just to clarify that all current xsk tests are independent, there is no subtest at present, and do not need to run in any order. Thanks, /Weqaar > > > >>>> + > >>>> +test_exit $retval 0 > >>>> diff --git a/tools/testing/selftests/bpf/test_xsk_drv_teardown.sh b/tools/testing/selftests/bpf/test_xsk_drv_teardown.sh > >>> [...]
This patch set adds AF_XDP selftests based on veth to selftests/bpf. # Topology: # --------- # ----------- # _ | Process | _ # / ----------- \ # / | \ # / | \ # ----------- | ----------- # | Thread1 | | | Thread2 | # ----------- | ----------- # | | | # ----------- | ----------- # | xskX | | | xskY | # ----------- | ----------- # | | | # ----------- | ---------- # | vethX | --------- | vethY | # ----------- peer ---------- # | | | # namespaceX | namespaceY These selftests test AF_XDP SKB and Native/DRV modes using veth Virtual Ethernet interfaces. The test program contains two threads, each thread is single socket with a unique UMEM. It validates in-order packet delivery and packet content by sending packets to each other. Prerequisites setup by script test_xsk_prerequisites.sh: Set up veth interfaces as per the topology shown ^^: * setup two veth interfaces and one namespace ** veth<xxxx> in root namespace ** veth<yyyy> in af_xdp<xxxx> namespace ** namespace af_xdp<xxxx> * create a spec file veth.spec that includes this run-time configuration that is read by test scripts - filenames prefixed with test_xsk_ *** xxxx and yyyy are randomly generated 4 digit numbers used to avoid conflict with any existing interface The following tests are provided: 1. AF_XDP SKB mode Generic mode XDP is driver independent, used when the driver does not have support for XDP. Works on any netdevice using sockets and generic XDP path. XDP hook from netif_receive_skb(). a. nopoll - soft-irq processing b. poll - using poll() syscall c. Socket Teardown Create a Tx and a Rx socket, Tx from one socket, Rx on another. Destroy both sockets, then repeat multiple times. Only nopoll mode is used d. Bi-directional Sockets Configure sockets as bi-directional tx/rx sockets, sets up fill and completion rings on each socket, tx/rx in both directions. Only nopoll mode is used 2. AF_XDP DRV/Native mode Works on any netdevice with XDP_REDIRECT support, driver dependent. Processes packets before SKB allocation. Provides better performance than SKB. Driver hook available just after DMA of buffer descriptor. a. nopoll b. poll c. Socket Teardown d. Bi-directional Sockets * Only copy mode is supported because veth does not currently support zero-copy mode Total tests: 8 Flow: * Single process spawns two threads: Tx and Rx * Each of these two threads attach to a veth interface within their assigned namespaces * Each thread creates one AF_XDP socket connected to a unique umem for each veth interface * Tx thread transmits 10k packets from veth<xxxx> to veth<yyyy> * Rx thread verifies if all 10k packets were received and delivered in-order, and have the right content v2 changes: * Move selftests/xsk to selftests/bpf * Remove Makefiles under selftests/xsk, and utilize selftests/bpf/Makefile Structure of the patch set: Patch 1: This patch adds XSK Selftests framework under selftests/bpf Patch 2: Adds tests: SKB poll and nopoll mode, and mac-ip-udp debug Patch 3: Adds tests: DRV poll and nopoll mode Patch 4: Adds tests: SKB and DRV Socket Teardown Patch 5: Adds tests: SKB and DRV Bi-directional Sockets Thanks: Weqaar Weqaar Janjua (5): selftests/bpf: xsk selftests framework selftests/bpf: xsk selftests - SKB POLL, NOPOLL selftests/bpf: xsk selftests - DRV POLL, NOPOLL selftests/bpf: xsk selftests - Socket Teardown - SKB, DRV selftests/bpf: xsk selftests - Bi-directional Sockets - SKB, DRV tools/testing/selftests/bpf/Makefile | 15 +- .../bpf/test_xsk_drv_bidirectional.sh | 23 + .../selftests/bpf/test_xsk_drv_nopoll.sh | 20 + .../selftests/bpf/test_xsk_drv_poll.sh | 20 + .../selftests/bpf/test_xsk_drv_teardown.sh | 20 + .../selftests/bpf/test_xsk_prerequisites.sh | 127 ++ .../bpf/test_xsk_skb_bidirectional.sh | 20 + .../selftests/bpf/test_xsk_skb_nopoll.sh | 20 + .../selftests/bpf/test_xsk_skb_poll.sh | 20 + .../selftests/bpf/test_xsk_skb_teardown.sh | 20 + tools/testing/selftests/bpf/xdpxceiver.c | 1056 +++++++++++++++++ tools/testing/selftests/bpf/xdpxceiver.h | 158 +++ tools/testing/selftests/bpf/xsk_env.sh | 28 + tools/testing/selftests/bpf/xsk_prereqs.sh | 119 ++ 14 files changed, 1664 insertions(+), 2 deletions(-) create mode 100755 tools/testing/selftests/bpf/test_xsk_drv_bidirectional.sh create mode 100755 tools/testing/selftests/bpf/test_xsk_drv_nopoll.sh create mode 100755 tools/testing/selftests/bpf/test_xsk_drv_poll.sh create mode 100755 tools/testing/selftests/bpf/test_xsk_drv_teardown.sh create mode 100755 tools/testing/selftests/bpf/test_xsk_prerequisites.sh create mode 100755 tools/testing/selftests/bpf/test_xsk_skb_bidirectional.sh create mode 100755 tools/testing/selftests/bpf/test_xsk_skb_nopoll.sh create mode 100755 tools/testing/selftests/bpf/test_xsk_skb_poll.sh create mode 100755 tools/testing/selftests/bpf/test_xsk_skb_teardown.sh create mode 100644 tools/testing/selftests/bpf/xdpxceiver.c create mode 100644 tools/testing/selftests/bpf/xdpxceiver.h create mode 100755 tools/testing/selftests/bpf/xsk_env.sh create mode 100755 tools/testing/selftests/bpf/xsk_prereqs.sh