mbox series

[net,0/4] selftests: mptcp: mark unstable subtests as flaky

Message ID 20240524-upstream-net-20240524-selftests-mptcp-flaky-v1-0-a352362f3f8e@kernel.org
Headers show
Series selftests: mptcp: mark unstable subtests as flaky | expand

Message

Matthieu Baerts (NGI0) May 24, 2024, 4:30 p.m. UTC
Some subtests can be unstable, failing once every X runs. Fixing them
can take time: there could be an issue in the kernel or in the subtest,
and it is then important to do a proper analysis, not to hide real bugs.

To avoid creating noises on the different CIs where tests are more
unstable than on our side, some subtests have been marked as flaky. As a
result, errors with these subtests (if any) are ignored.

Note that the MPTCP CI will continue to track these flaky subtests. All
these unstable subtests are also tracked by our bug tracker.

These are fixes for the -net tree, because the instabilities are visible
there. The first patch introducing the flake support has no 'Fixes'
tags, mainly because it requires recent and important refactoring done
in all MPTCP selftests. Backporting that to old versions where the flaky
tests have been introduced would be too difficult, and probably not
worth it. The other patches, adding MPTCP_LIB_SUBTEST_FLAKY=1, have a
Fixes tag, simply to ease the backport of the future fixes removing them
along with the proper fix.

Signed-off-by: Matthieu Baerts (NGI0) <matttbe@kernel.org>
---
Matthieu Baerts (NGI0) (4):
      selftests: mptcp: lib: support flaky subtests
      selftests: mptcp: simult flows: mark 'unbalanced' tests as flaky
      selftests: mptcp: join: mark 'fastclose' tests as flaky
      selftests: mptcp: join: mark 'fail' tests as flaky

 tools/testing/selftests/net/mptcp/mptcp_join.sh   | 10 +++++++-
 tools/testing/selftests/net/mptcp/mptcp_lib.sh    | 30 +++++++++++++++++++++--
 tools/testing/selftests/net/mptcp/simult_flows.sh |  6 ++---
 3 files changed, 40 insertions(+), 6 deletions(-)
---
base-commit: 0b4f5add9fa59bfd42c1030f572db2e4c395181b
change-id: 20240524-upstream-net-20240524-selftests-mptcp-flaky-5b6836a59f72

Best regards,

Comments

patchwork-bot+netdevbpf@kernel.org May 28, 2024, 12:20 a.m. UTC | #1
Hello:

This series was applied to netdev/net.git (main)
by Jakub Kicinski <kuba@kernel.org>:

On Fri, 24 May 2024 18:30:55 +0200 you wrote:
> Some subtests can be unstable, failing once every X runs. Fixing them
> can take time: there could be an issue in the kernel or in the subtest,
> and it is then important to do a proper analysis, not to hide real bugs.
> 
> To avoid creating noises on the different CIs where tests are more
> unstable than on our side, some subtests have been marked as flaky. As a
> result, errors with these subtests (if any) are ignored.
> 
> [...]

Here is the summary with links:
  - [net,1/4] selftests: mptcp: lib: support flaky subtests
    https://git.kernel.org/netdev/net/c/5597613fb3cf
  - [net,2/4] selftests: mptcp: simult flows: mark 'unbalanced' tests as flaky
    https://git.kernel.org/netdev/net/c/cc73a6577ae6
  - [net,3/4] selftests: mptcp: join: mark 'fastclose' tests as flaky
    https://git.kernel.org/netdev/net/c/8c06ac2178a9
  - [net,4/4] selftests: mptcp: join: mark 'fail' tests as flaky
    https://git.kernel.org/netdev/net/c/38af56e6668b

You are awesome, thank you!