From patchwork Wed Nov 15 17:43:50 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nikolai Kondrashov X-Patchwork-Id: 744292 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="akYvoXPN" Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1DB8D1B8 for ; Wed, 15 Nov 2023 09:52:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1700070741; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=zkma+ytFECSmUGaVR3H0P8/excTRmPEEewT1e65f88Q=; b=akYvoXPN8A7XHK9ZBSBA0L43/Dz98fs/a3rL8JujbAAlD35Jj0VfHJOMharxnCUqrAYVgZ MsXybdPck+LzEq8eR3U/jtH03osRlz2EY3mgqVGIoGpJk77SY+sBl5uB8gH/nAvzaeidb9 M+jUPqaY2bnbYFo5/ElwAWQU6XHbCkY= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-373-X-IRN8qTPlG_KX3opGHxDA-1; Wed, 15 Nov 2023 12:52:18 -0500 X-MC-Unique: X-IRN8qTPlG_KX3opGHxDA-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 760498007B3; Wed, 15 Nov 2023 17:52:17 +0000 (UTC) Received: from ruby.redhat.com (unknown [10.39.192.56]) by smtp.corp.redhat.com (Postfix) with ESMTP id 0FB741121306; Wed, 15 Nov 2023 17:52:14 +0000 (UTC) From: Nikolai Kondrashov To: workflows@vger.kernel.org, Joe Perches , Andy Whitcroft , Theodore Ts'o , David Gow , Steven Rostedt , Mark Brown , Shuah Khan , "Darrick J . Wong" Cc: kunit-dev@googlegroups.com, linux-kselftest@vger.kernel.org, Veronika Kabatova , CKI , kernelci@lists.linux.dev, Nikolai Kondrashov Subject: [PATCH 2/3] MAINTAINERS: Require kvm-xfstests smoke for ext4 Date: Wed, 15 Nov 2023 19:43:50 +0200 Message-ID: <20231115175146.9848-3-Nikolai.Kondrashov@redhat.com> In-Reply-To: <20231115175146.9848-1-Nikolai.Kondrashov@redhat.com> References: <20231115175146.9848-1-Nikolai.Kondrashov@redhat.com> Precedence: bulk X-Mailing-List: linux-kselftest@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.3 Signed-off-by: Nikolai Kondrashov --- Documentation/process/tests.rst | 32 ++++++++++++++++++++++++++++++++ MAINTAINERS | 1 + 2 files changed, 33 insertions(+) diff --git a/Documentation/process/tests.rst b/Documentation/process/tests.rst index 907311e91ec45..9a9ea3fe65c37 100644 --- a/Documentation/process/tests.rst +++ b/Documentation/process/tests.rst @@ -33,3 +33,35 @@ particularly useful: :Source: A URL pointing to the source code of the test suite :Docs: A URL pointing to further test suite documentation + +xfstests +-------- + +:Summary: File system regression test suite +:Source: git://git.kernel.org/pub/scm/fs/xfs/xfstests-dev.git +:Docs: https://github.com/tytso/xfstests-bld/blob/master/Documentation/what-is-xfstests.md + +As the name might imply, xfstests is a file system regression test suite which +was originally developed by Silicon Graphics (SGI) for the XFS file system. +Originally, xfstests, like XFS was only supported on the SGI's Irix operating +system. When XFS was ported to Linux, so was xfstests, and now xfstests is +only supported on Linux. + +Today, xfstests is used as a file system regression test suite for all of +Linux's major file systems: xfs, ext2, ext4, cifs, btrfs, f2fs, reiserfs, gfs, +jfs, udf, nfs, and tmpfs. Many file system maintainers will run a full set of +xfstests before sending patches to Linus, and will require that any major +changes be tested using xfstests before they are submitted for integration. + +The easiest way to start running xfstests is under KVM with xfstests-bld: +https://github.com/tytso/xfstests-bld/blob/master/Documentation/kvm-quickstart.md + +kvm-xfstests smoke +------------------ + +:Summary: File system smoke tests +:Superset: xfstests +:Docs: https://github.com/tytso/xfstests-bld/blob/master/Documentation/kvm-quickstart.md + +The "kvm-xfstests smoke" is a minimal subset of xfstests for testing all major +file systems, running under KVM. diff --git a/MAINTAINERS b/MAINTAINERS index 2565c04f0490e..f81a47d87ac26 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -7974,6 +7974,7 @@ L: linux-ext4@vger.kernel.org S: Maintained W: http://ext4.wiki.kernel.org Q: http://patchwork.ozlabs.org/project/linux-ext4/list/ +V: kvm-xfstests smoke T: git git://git.kernel.org/pub/scm/linux/kernel/git/tytso/ext4.git F: Documentation/filesystems/ext4/ F: fs/ext4/ From patchwork Wed Nov 15 17:43:51 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nikolai Kondrashov X-Patchwork-Id: 744291 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="Sag0OzX5" Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6E422127 for ; Wed, 15 Nov 2023 09:52:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1700070743; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=OG/PdxLPzgrb2SSqLq85wNnPv1wHigzKlm1PKA/CjnI=; b=Sag0OzX5Nz3cSUqbAdodwmnbvTTOlKGR89n+VTXmyJ1AEKdtMfmPtFRxCXsn9E6bJ3iOk9 d9jrlYV8yTPNE196CcVSIpDlcT6gQsJrfLA1rBSSM4uTqlGtgnOdySKOP9AhDPikpz0MO+ dbNa3WMBbveUAGCaOGS7+tpOUacvVRk= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-438-1_fhHc8YNcmXv39uhnoUYw-1; Wed, 15 Nov 2023 12:52:21 -0500 X-MC-Unique: 1_fhHc8YNcmXv39uhnoUYw-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id C039D834FC6; Wed, 15 Nov 2023 17:52:20 +0000 (UTC) Received: from ruby.redhat.com (unknown [10.39.192.56]) by smtp.corp.redhat.com (Postfix) with ESMTP id B96FA1121306; Wed, 15 Nov 2023 17:52:17 +0000 (UTC) From: Nikolai Kondrashov To: workflows@vger.kernel.org, Joe Perches , Andy Whitcroft , Theodore Ts'o , David Gow , Steven Rostedt , Mark Brown , Shuah Khan , "Darrick J . Wong" Cc: kunit-dev@googlegroups.com, linux-kselftest@vger.kernel.org, Veronika Kabatova , CKI , kernelci@lists.linux.dev, Nikolai Kondrashov Subject: [PATCH 3/3] MAINTAINERS: Require kunit core tests for framework changes Date: Wed, 15 Nov 2023 19:43:51 +0200 Message-ID: <20231115175146.9848-4-Nikolai.Kondrashov@redhat.com> In-Reply-To: <20231115175146.9848-1-Nikolai.Kondrashov@redhat.com> References: <20231115175146.9848-1-Nikolai.Kondrashov@redhat.com> Precedence: bulk X-Mailing-List: linux-kselftest@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.3 Signed-off-by: Nikolai Kondrashov --- Documentation/process/tests.rst | 13 +++++++++++++ MAINTAINERS | 1 + 2 files changed, 14 insertions(+) diff --git a/Documentation/process/tests.rst b/Documentation/process/tests.rst index 9a9ea3fe65c37..56a7911f69031 100644 --- a/Documentation/process/tests.rst +++ b/Documentation/process/tests.rst @@ -65,3 +65,16 @@ kvm-xfstests smoke The "kvm-xfstests smoke" is a minimal subset of xfstests for testing all major file systems, running under KVM. + +kunit +----- + +:Summary: The complete set of KUnit unit tests +:Command: tools/testing/kunit/kunit.py run --alltests + +kunit core +---------- + +:Summary: KUnit tests for the framework itself +:Superset: kunit +:Command: tools/testing/kunit/kunit.py run --kunitconfig lib/kunit diff --git a/MAINTAINERS b/MAINTAINERS index f81a47d87ac26..5f3261e96c90f 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -11626,6 +11626,7 @@ L: linux-kselftest@vger.kernel.org L: kunit-dev@googlegroups.com S: Maintained W: https://google.github.io/kunit-docs/third_party/kernel/docs/ +V: kunit core T: git git://git.kernel.org/pub/scm/linux/kernel/git/shuah/linux-kselftest.git kunit T: git git://git.kernel.org/pub/scm/linux/kernel/git/shuah/linux-kselftest.git kunit-fixes F: Documentation/dev-tools/kunit/ From patchwork Tue Dec 5 18:03:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nikolai Kondrashov X-Patchwork-Id: 750559 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="MECBrpVu" Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 63BEC1BD for ; Tue, 5 Dec 2023 10:45:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1701801948; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=EQBrrsVFSJUf5yblNEzw3jtgF+8KfPMfViCDC3QAM68=; b=MECBrpVuv3StiumtIvB7ATXtXiJ1MgWTIb3khaisDDxbdfc2BVaeM/IhX6hkCKV9rUvWfU c9PIf1fEst/ZkzOgsGXZYsnBgMGzhfclfrd3QPxMZqELCKrlJzjdsrTalZmhVSSy5f5vG7 NrAC6Cn8WSv/FcqmC3A+9egiT86ZPpE= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-90-p7DFPLn-PrqHkCgP5_2T_w-1; Tue, 05 Dec 2023 13:45:45 -0500 X-MC-Unique: p7DFPLn-PrqHkCgP5_2T_w-1 Received: from smtp.corp.redhat.com (int-mx10.intmail.prod.int.rdu2.redhat.com [10.11.54.10]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 8712788B790; Tue, 5 Dec 2023 18:45:44 +0000 (UTC) Received: from ruby.redhat.com (unknown [10.39.192.2]) by smtp.corp.redhat.com (Postfix) with ESMTP id EEEFD492BE6; Tue, 5 Dec 2023 18:45:41 +0000 (UTC) From: Nikolai Kondrashov To: workflows@vger.kernel.org, Jonathan Corbet , Joe Perches , Andy Whitcroft , Theodore Ts'o , David Gow , Steven Rostedt , Mark Brown , Shuah Khan , "Darrick J . Wong" Cc: kunit-dev@googlegroups.com, linux-kselftest@vger.kernel.org, Veronika Kabatova , CKI , kernelci@lists.linux.dev, Nikolai Kondrashov Subject: [RFC PATCH v2 04/10] docs: submitting-patches: Introduce Tested-with: Date: Tue, 5 Dec 2023 20:03:00 +0200 Message-ID: <20231205184503.79769-5-Nikolai.Kondrashov@redhat.com> In-Reply-To: <20231205184503.79769-1-Nikolai.Kondrashov@redhat.com> References: <20231115175146.9848-1-Nikolai.Kondrashov@redhat.com> <20231205184503.79769-1-Nikolai.Kondrashov@redhat.com> Precedence: bulk X-Mailing-List: linux-kselftest@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.10 Introduce a new tag, 'Tested-with:', documented in the Documentation/process/submitting-patches.rst file. The tag is expected to contain the test suite command which was executed for the commit, and to certify it passed. Additionally, it can contain a URL pointing to the execution results, after a '#' character. Prohibit the V: field from containing the '#' character correspondingly. Signed-off-by: Nikolai Kondrashov --- Documentation/process/submitting-patches.rst | 10 ++++++++++ MAINTAINERS | 2 +- scripts/checkpatch.pl | 4 ++-- 3 files changed, 13 insertions(+), 3 deletions(-) diff --git a/Documentation/process/submitting-patches.rst b/Documentation/process/submitting-patches.rst index f034feaf1369e..2004df2ac1b39 100644 --- a/Documentation/process/submitting-patches.rst +++ b/Documentation/process/submitting-patches.rst @@ -245,6 +245,16 @@ Execute the commands, if any, to test your changes. All commands must be executed from the root of the source tree. Each command outputs usage information, if an -h/--help option is specified. +If a test suite you've executed completed successfully, add a ``Tested-with: +`` to the message of the commit you tested. E.g.:: + + Tested-with: tools/testing/kunit/run_checks.py + +Optionally, add a '#' character followed by a publicly-accessible URL +containing the test results, if you make them available. E.g.:: + + Tested-with: tools/testing/kunit/run_checks.py # https://kernelci.org/test/2239874 + Select the recipients for your patch ------------------------------------ diff --git a/MAINTAINERS b/MAINTAINERS index 68821eecf61cf..28fbb0eb335ba 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -63,7 +63,7 @@ Descriptions of section entries and preferred order executed to verify changes to the maintained subsystem. Must be executed from the root of the source tree. Must support the -h/--help option. - Cannot contain '@' character. + Cannot contain '@' or '#' characters. V: tools/testing/kunit/run_checks.py One test suite per line. diff --git a/scripts/checkpatch.pl b/scripts/checkpatch.pl index a184e576c980b..bea602c30df5d 100755 --- a/scripts/checkpatch.pl +++ b/scripts/checkpatch.pl @@ -3686,9 +3686,9 @@ sub process { # check MAINTAINERS V: entries are valid if ($rawline =~ /^\+V:\s*(.*)/) { my $name = $1; - if ($name =~ /@/) { + if ($name =~ /[@#]/) { ERROR("TEST_PROPOSAL_INVALID", - "Test proposal cannot contain '\@' character\n" . $herecurr); + "Test proposal cannot contain '\@' or '#' characters\n" . $herecurr); } } } From patchwork Tue Dec 5 18:03:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nikolai Kondrashov X-Patchwork-Id: 750558 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="geMsWfyZ" Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9BF8B194 for ; Tue, 5 Dec 2023 10:45:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1701801954; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=D/3FU+psVxeFvo/nAcRzqPHgzPOCqZqJqnGBfx/xpzA=; b=geMsWfyZZU7cOJutiq+jew+Wv7d0wWd6LSuYx69A0q0KTtoS+mfafGG6qRYz/Coba/YMOI jkg75TDVqScpkBLeskC5RPLkXyEIBwnUFAJSGJKveRRPK0IVB6mGK2P29VOeisg4bj/RLF UJUFLzG1G793KoTzHXFetwsRnRZL5Ws= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-426-oKcDfZUYP92HLE1XibBcBg-1; Tue, 05 Dec 2023 13:45:51 -0500 X-MC-Unique: oKcDfZUYP92HLE1XibBcBg-1 Received: from smtp.corp.redhat.com (int-mx10.intmail.prod.int.rdu2.redhat.com [10.11.54.10]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 40F95102F00D; Tue, 5 Dec 2023 18:45:50 +0000 (UTC) Received: from ruby.redhat.com (unknown [10.39.192.2]) by smtp.corp.redhat.com (Postfix) with ESMTP id A5A08492BE6; Tue, 5 Dec 2023 18:45:47 +0000 (UTC) From: Nikolai Kondrashov To: workflows@vger.kernel.org, Jonathan Corbet , Joe Perches , Andy Whitcroft , Theodore Ts'o , David Gow , Steven Rostedt , Mark Brown , Shuah Khan , "Darrick J . Wong" Cc: kunit-dev@googlegroups.com, linux-kselftest@vger.kernel.org, Veronika Kabatova , CKI , kernelci@lists.linux.dev, Nikolai Kondrashov Subject: [RFC PATCH v2 06/10] MAINTAINERS: Support referencing test docs in V: Date: Tue, 5 Dec 2023 20:03:02 +0200 Message-ID: <20231205184503.79769-7-Nikolai.Kondrashov@redhat.com> In-Reply-To: <20231205184503.79769-1-Nikolai.Kondrashov@redhat.com> References: <20231115175146.9848-1-Nikolai.Kondrashov@redhat.com> <20231205184503.79769-1-Nikolai.Kondrashov@redhat.com> Precedence: bulk X-Mailing-List: linux-kselftest@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.10 Support referencing test suite documentation in the V: entries of MAINTAINERS file. Use the '*' syntax (like C pointer dereference), where '' is a second-level heading in the new Documentation/process/tests.rst file, with the suite's description. This syntax allows distinguishing the references from test commands. Add a boiler-plate Documentation/process/tests.rst file, describing a way to add structured info to the test suites in the form of field lists. Apart from a "summary" and "command" fields, they can also contain a "superset" field specifying the superset of the test suite, helping reuse documentation and express both wider and narrower test sets. Make scripts/checkpatch.pl load the tests from the file, along with the structured data, validate the references in MAINTAINERS, dereference them, and output the test suite information in the CHECK messages whenever the corresponding subsystems are changed. But only if there was no corresponding Tested-with: tag in the commit message, certifying it was executed successfully already. This is supposed to help propose executing test suites which cannot be executed immediately, and need extra setup, as well as provide a place for extra documentation and information on directly-available suites. Signed-off-by: Nikolai Kondrashov --- Documentation/process/index.rst | 1 + Documentation/process/submitting-patches.rst | 21 +++- Documentation/process/tests.rst | 41 +++++++ MAINTAINERS | 9 +- scripts/checkpatch.pl | 122 +++++++++++++++++-- 5 files changed, 177 insertions(+), 17 deletions(-) create mode 100644 Documentation/process/tests.rst diff --git a/Documentation/process/index.rst b/Documentation/process/index.rst index a1daa309b58d0..3eda2e7432fdb 100644 --- a/Documentation/process/index.rst +++ b/Documentation/process/index.rst @@ -49,6 +49,7 @@ Other guides to the community that are of interest to most developers are: :maxdepth: 1 changes + tests stable-api-nonsense management-style stable-kernel-rules diff --git a/Documentation/process/submitting-patches.rst b/Documentation/process/submitting-patches.rst index 2004df2ac1b39..45bd1a713ef33 100644 --- a/Documentation/process/submitting-patches.rst +++ b/Documentation/process/submitting-patches.rst @@ -233,27 +233,42 @@ Test your changes Test the patch to the best of your ability. Check the MAINTAINERS file for the subsystem(s) you are changing to see if there are any **V:** entries -proposing particular test suite commands. E.g.:: +proposing particular test suites, either directly as commands, or via +documentation references. + +Test suite references start with a ``*`` (similar to C pointer dereferencing), +followed by the name of the test suite, which would be documented in the +Documentation/process/tests.rst under the corresponding heading. E.g.:: + + V: *xfstests + +Anything not starting with a ``*`` is considered a command. E.g.:: V: tools/testing/kunit/run_checks.py Supplying the ``--test`` option to ``scripts/get_maintainer.pl`` adds **V:** entries to its output. -Execute the commands, if any, to test your changes. +Execute the (referenced) test suites, if any, to test your changes. All commands must be executed from the root of the source tree. Each command outputs usage information, if an -h/--help option is specified. If a test suite you've executed completed successfully, add a ``Tested-with: -`` to the message of the commit you tested. E.g.:: +`` or ``Tested-with: *`` to the message of the commit you +tested. E.g.:: Tested-with: tools/testing/kunit/run_checks.py +or:: + + Tested-with: *xfstests + Optionally, add a '#' character followed by a publicly-accessible URL containing the test results, if you make them available. E.g.:: Tested-with: tools/testing/kunit/run_checks.py # https://kernelci.org/test/2239874 + Tested-with: *xfstests # https://kernelci.org/test/2239324 Select the recipients for your patch diff --git a/Documentation/process/tests.rst b/Documentation/process/tests.rst new file mode 100644 index 0000000000000..4ae5000e811c8 --- /dev/null +++ b/Documentation/process/tests.rst @@ -0,0 +1,41 @@ +.. SPDX-License-Identifier: GPL-2.0 + +.. _tests: + +Tests you can run +================= + +There are many automated tests available for the Linux kernel, and some +userspace tests which happen to also test the kernel. Here are some of them, +along with the instructions on where to get them and how to run them for +various purposes. + +This document has to follow a certain structure to allow tool access. +Second-level headers (underscored with dashes '-') must contain test suite +names, and the corresponding section must contain the test description. + +The test suites can be referenced by name, preceded with a '*', in the "V:" +lines in the MAINTAINERS file, as well as in the "Tested-with:" tag in commit +messages. E.g:: + + V: *xfstests + +and:: + + Tested-with: *xfstests + +Additionally, test suite names cannot contain '@' or '#' characters, the same +as "V:" entries. + +The test suite description should contain the test documentation in general: +where to get the test, how to run it, and how to interpret its results, but +could also start with a "field list", containing single-line entries, with the +following ones recognized by the tools (regardless of the case): + +:Summary: a single-line summary of the test suite (singular, non-capitalized) +:Superset: the name of the test suite this one is a subset of +:Command: a shell command executing the test suite, not requiring setup + beyond the kernel tree and regular developer environment + (even if only to report what else needs setting up) + +Any other entries are accepted, but not processed. diff --git a/MAINTAINERS b/MAINTAINERS index 28fbb0eb335ba..3ed15d8327919 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -60,11 +60,14 @@ Descriptions of section entries and preferred order printk, pr_info or pr_err One regex pattern per line. Multiple K: lines acceptable. V: *Test suite* proposed for execution. The command that could be - executed to verify changes to the maintained subsystem. - Must be executed from the root of the source tree. - Must support the -h/--help option. + executed to verify changes to the maintained subsystem, or a reference + to a test suite documented in Documentation/process/tests.txt. + Commands must be executed from the root of the source tree. + Commands must support the -h/--help option. + References must be preceded with a '*'. Cannot contain '@' or '#' characters. V: tools/testing/kunit/run_checks.py + V: *xfstests One test suite per line. Maintainers List diff --git a/scripts/checkpatch.pl b/scripts/checkpatch.pl index 1da617e1edb5f..bfeb4c33b5424 100755 --- a/scripts/checkpatch.pl +++ b/scripts/checkpatch.pl @@ -66,6 +66,9 @@ my $codespellfile = "/usr/share/codespell/dictionary.txt"; my $user_codespellfile = ""; my $conststructsfile = "$D/const_structs.checkpatch"; my $docsfile = "$D/../Documentation/dev-tools/checkpatch.rst"; +my $testsrelfile = "Documentation/process/tests.rst"; +my $testsfile = "$D/../$testsrelfile"; +my %tests = (); my $typedefsfile; my $color = "auto"; my $allow_c99_comments = 1; # Can be overridden by --ignore C99_COMMENT_TOLERANCE @@ -282,6 +285,39 @@ sub load_docs { close($docs); } +sub load_tests { + open(my $tests, '<', "$testsfile") + or warn "$P: Can't read the tests file $testsfile $!\n"; + + my $name = undef; + my $prev_line = undef; + my $in_field_list = 0; + + while (<$tests>) { + my $line = $_; + $line =~ s/\s+$//; + + # If the previous line was a second-level header (test name) + if ($line =~ /^-+$/ && + defined($prev_line) && + length($line) == length($prev_line)) { + $name = $prev_line; + $tests{$name} = {}; + $in_field_list = 1; + # Else, if we're parsing the test's header field list + } elsif ($in_field_list) { + if ($line =~ /^:([^:]+):\s+(.*)/) { + $tests{$name}{lc($1)} = $2; + } else { + $in_field_list = !$line; + } + } + + $prev_line = $line; + } + close($tests); +} + # Perl's Getopt::Long allows options to take optional arguments after a space. # Prevent --color by itself from consuming other arguments foreach (@ARGV) { @@ -372,6 +408,7 @@ if ($color =~ /^[01]$/) { load_docs() if ($verbose); list_types(0) if ($list_types); +load_tests(); $fix = 1 if ($fix_inplace); $check_orig = $check; @@ -1160,10 +1197,22 @@ sub get_file_proposed_tests { my $output = `$command 2>/dev/null`; # But regenerate stderr on failure die "Failed retrieving tests proposed for changes to \"$filename\":\n" . `$command 2>&1 >/dev/null` if ($?); - $files_proposed_tests{$filename} = [grep { !/@/ } split("\n", $output)] + $file_proposed_tests = [grep { !/@/ } split("\n", $output)]; + # Validate and normalize all references + for my $index (0 .. scalar @$file_proposed_tests - 1) { + my $test = $file_proposed_tests->[$index]; + if ($test =~ /^\*\s*(.*)$/) { + my $name = $1; + die "Test $name referenced in MAINTAINERS not found in $testsrelfile\n" + if (!exists $tests{$name}); + $file_proposed_tests->[$index] = "*" . $name; + } + } + $files_proposed_tests{$filename} = $file_proposed_tests; + } else { + $file_proposed_tests = $files_proposed_tests{$filename}; } - $file_proposed_tests = $files_proposed_tests{$filename}; return @$file_proposed_tests; } @@ -2936,11 +2985,33 @@ sub process { # Check if tests are proposed for changes to the file foreach my $test (get_file_proposed_tests($realfile)) { next if exists $dont_propose_tests{$test}; - CHK("TEST_PROPOSAL", - "Running the following test suite is proposed for changes to $realfile:\n" . - "$test\n" . - "Add the following to the tested commit's message, IF IT PASSES:\n" . - "Tested-with: $test\n"); + my $name; + my $title; + my $command; + my $message; + # If this is a reference to a documented test suite + if ($test =~ /^\*\s*(.*)/) { + $name = $1; + $title = $tests{$name}{"summary"} // "$name test suite"; + $command = $tests{$name}{"command"}; + # Else it's a test command + } else { + $title = "test suite"; + $command = $test; + } + if ($command) { + $message = "Execute the $title " . + "proposed for verifying changes to $realfile:\n" . + "$command\n"; + } else { + $message = "The $title is proposed for verifying changes to $realfile\n"; + } + if ($name) { + $message .= "See instructions under \"$name\" in $testsrelfile\n"; + } + $message .= "Add the following to the tested commit's message, " . + "IF IT PASSES:\nTested-with: $test\n"; + CHK("TEST_PROPOSAL", $message); $dont_propose_tests{$test} = 1; } @@ -3272,8 +3343,28 @@ sub process { # Check and accumulate executed test suites (stripping URLs off the end) if (!$in_commit_log && $line =~ /^\s*Tested-with:\s*(.*?)\s*#.*$/i) { - # Do not propose this certified-passing test suite - $dont_propose_tests{$1} = 1; + my $test = $1; + # If the test is a reference + if ($test =~ /^\*\s*(.*)$/) { + # Do not propose (normalized references to) + # the test and its subsets + local *dont_propose_test_name = sub { + my ($name) = @_; + $dont_propose_tests{"*" . $name} = 1; + foreach my $sub_name (keys %tests) { + my $sub_data = $tests{$sub_name}; + my $superset = $sub_data->{"superset"}; + if (defined($superset) and $superset eq $name) { + dont_propose_test($sub_name); + } + } + }; + dont_propose_test_name($1); + # Else it's a command + } else { + # Do not propose the test + $dont_propose_tests{$test} = 1; + } } # Check email subject for common tools that don't need to be mentioned @@ -3728,8 +3819,17 @@ sub process { } # check MAINTAINERS V: entries are valid if ($rawline =~ /^\+V:\s*(.*)/) { - my $name = $1; - if ($name =~ /[@#]/) { + my $entry = $1; + # If this is a valid entry value + if ($entry =~ /^[^@#]*$/) { + # If the test in the entry is a reference + if ($entry =~ /^\*\s*(.*)$/) { + my $name = $1; + ERROR("TEST_PROPOSAL_INVALID", + "Test $name referenced in MAINTAINERS not found in $testsrelfile\n" . + $herecurr) if (!exists $tests{$name}); + } + } else { ERROR("TEST_PROPOSAL_INVALID", "Test proposal cannot contain '\@' or '#' characters\n" . $herecurr); } From patchwork Tue Dec 5 18:03:04 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nikolai Kondrashov X-Patchwork-Id: 750557 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="EPBywQkG" Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 40AD8BA for ; Tue, 5 Dec 2023 10:46:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1701801960; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=mE8ijlql5yJARnazpyl45/u3+npm3s09B/RpWnHdyUU=; b=EPBywQkGcnqtrRE8QYnMzIXYYBFyHyTxbXEQZvZ5zhqGQAtOFK85kgFz22LJVdQsq5MHkm dHgiKM0y0LAgGwDheBq2MedSsOMlaFd2htM/L6fpjn1mzdblhbXu1dpk13/MGEE2vALoRS gYKxFjHJ2h+GZBrJq3Mgmq6yTtgjJ9Y= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-126-CgPwecmMMKuMmYV-O7r4jg-1; Tue, 05 Dec 2023 13:45:56 -0500 X-MC-Unique: CgPwecmMMKuMmYV-O7r4jg-1 Received: from smtp.corp.redhat.com (int-mx10.intmail.prod.int.rdu2.redhat.com [10.11.54.10]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 11BB51C068EC; Tue, 5 Dec 2023 18:45:56 +0000 (UTC) Received: from ruby.redhat.com (unknown [10.39.192.2]) by smtp.corp.redhat.com (Postfix) with ESMTP id 7539F492BE6; Tue, 5 Dec 2023 18:45:53 +0000 (UTC) From: Nikolai Kondrashov To: workflows@vger.kernel.org, Jonathan Corbet , Joe Perches , Andy Whitcroft , Theodore Ts'o , David Gow , Steven Rostedt , Mark Brown , Shuah Khan , "Darrick J . Wong" Cc: kunit-dev@googlegroups.com, linux-kselftest@vger.kernel.org, Veronika Kabatova , CKI , kernelci@lists.linux.dev, Nikolai Kondrashov Subject: [RFC PATCH v2 08/10] docs: tests: Document kunit in general Date: Tue, 5 Dec 2023 20:03:04 +0200 Message-ID: <20231205184503.79769-9-Nikolai.Kondrashov@redhat.com> In-Reply-To: <20231205184503.79769-1-Nikolai.Kondrashov@redhat.com> References: <20231115175146.9848-1-Nikolai.Kondrashov@redhat.com> <20231205184503.79769-1-Nikolai.Kondrashov@redhat.com> Precedence: bulk X-Mailing-List: linux-kselftest@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.10 Add an entry on the complete set of kunit tests to the Documentation/process/tests.rst, so that it could be referenced in MAINTAINERS, and is catalogued in general. Signed-off-by: Nikolai Kondrashov --- Documentation/process/tests.rst | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/Documentation/process/tests.rst b/Documentation/process/tests.rst index cfaf937dc4d5f..0760229fc32b0 100644 --- a/Documentation/process/tests.rst +++ b/Documentation/process/tests.rst @@ -71,3 +71,26 @@ kvm-xfstests smoke The "kvm-xfstests smoke" is a minimal subset of xfstests for testing all major file systems, running under KVM. + +kunit +----- + +:Summary: complete set of KUnit unit tests +:Command: tools/testing/kunit/kunit.py run --alltests +:Docs: https://docs.kernel.org/dev-tools/kunit/ + +KUnit tests are part of the kernel, written in the C (programming) language, +and test parts of the Kernel implementation (example: a C language function). +Excluding build time, from invocation to completion, KUnit can run around 100 +tests in less than 10 seconds. KUnit can test any kernel component, for +example: file system, system calls, memory management, device drivers and so +on. + +KUnit follows the white-box testing approach. The test has access to internal +system functionality. KUnit runs in kernel space and is not restricted to +things exposed to user-space. + +In addition, KUnit has kunit_tool, a script (tools/testing/kunit/kunit.py) +that configures the Linux kernel, runs KUnit tests under QEMU or UML (User +Mode Linux), parses the test results and displays them in a user friendly +manner. From patchwork Tue Dec 5 18:03:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nikolai Kondrashov X-Patchwork-Id: 750556 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="RuxDTTPc" Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 95E56C0 for ; Tue, 5 Dec 2023 10:46:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1701801965; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=CfSpnYy4oKsXzvE3S4cKH6fDl4J4nm+5m3oSRUCkKP4=; b=RuxDTTPcBF0tHtTEDleznNugqm1XXzHk4+UtjFFFtWrlXWEz418Ucz3JnolbKfzmIZ3np/ 9L4EzoFmAAU1hPlEQ6aRTQj1HfBTRiCgzv/tUUWy8L8fcprQJDv3Mmhnnq013Dgt4q1ril IZ9LdbBDYECOL4AKwGZFgjeN5CfKSDw= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-225-kxiglrEjPbC8QQ1GeR3YAw-1; Tue, 05 Dec 2023 13:46:02 -0500 X-MC-Unique: kxiglrEjPbC8QQ1GeR3YAw-1 Received: from smtp.corp.redhat.com (int-mx10.intmail.prod.int.rdu2.redhat.com [10.11.54.10]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 2B3DF185A785; Tue, 5 Dec 2023 18:46:02 +0000 (UTC) Received: from ruby.redhat.com (unknown [10.39.192.2]) by smtp.corp.redhat.com (Postfix) with ESMTP id 30955492BE6; Tue, 5 Dec 2023 18:45:59 +0000 (UTC) From: Nikolai Kondrashov To: workflows@vger.kernel.org, Jonathan Corbet , Joe Perches , Andy Whitcroft , Theodore Ts'o , David Gow , Steven Rostedt , Mark Brown , Shuah Khan , "Darrick J . Wong" Cc: kunit-dev@googlegroups.com, linux-kselftest@vger.kernel.org, Veronika Kabatova , CKI , kernelci@lists.linux.dev, Nikolai Kondrashov Subject: [RFC PATCH v2 10/10] MAINTAINERS: Add proposal strength to V: entries Date: Tue, 5 Dec 2023 20:03:06 +0200 Message-ID: <20231205184503.79769-11-Nikolai.Kondrashov@redhat.com> In-Reply-To: <20231205184503.79769-1-Nikolai.Kondrashov@redhat.com> References: <20231115175146.9848-1-Nikolai.Kondrashov@redhat.com> <20231205184503.79769-1-Nikolai.Kondrashov@redhat.com> Precedence: bulk X-Mailing-List: linux-kselftest@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.10 Require the MAINTAINERS V: entries to begin with a keyword, one of SUGGESTED/RECOMMENDED/REQUIRED, signifying how strongly the test is proposed for verifying the subsystem changes, prompting scripts/checkpatch.pl to produce CHECK/WARNING/ERROR messages respectively, whenever the commit message doesn't have the corresponding Tested-with: tag. Signed-off-by: Nikolai Kondrashov --- Documentation/process/submitting-patches.rst | 11 ++- MAINTAINERS | 20 +++-- scripts/checkpatch.pl | 83 ++++++++++++-------- 3 files changed, 71 insertions(+), 43 deletions(-) diff --git a/Documentation/process/submitting-patches.rst b/Documentation/process/submitting-patches.rst index 45bd1a713ef33..199fadc50cf62 100644 --- a/Documentation/process/submitting-patches.rst +++ b/Documentation/process/submitting-patches.rst @@ -233,18 +233,21 @@ Test your changes Test the patch to the best of your ability. Check the MAINTAINERS file for the subsystem(s) you are changing to see if there are any **V:** entries -proposing particular test suites, either directly as commands, or via -documentation references. +proposing particular test suites. + +The **V:** entries start with a proposal strength keyword +(SUGGESTED/RECOMMENDED/REQUIRED), followed either by a command, or a +documentation reference. Test suite references start with a ``*`` (similar to C pointer dereferencing), followed by the name of the test suite, which would be documented in the Documentation/process/tests.rst under the corresponding heading. E.g.:: - V: *xfstests + V: SUGGESTED *xfstests Anything not starting with a ``*`` is considered a command. E.g.:: - V: tools/testing/kunit/run_checks.py + V: RECOMMENDED tools/testing/kunit/run_checks.py Supplying the ``--test`` option to ``scripts/get_maintainer.pl`` adds **V:** entries to its output. diff --git a/MAINTAINERS b/MAINTAINERS index 84e90ec015090..3a35e320b5a5b 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -59,15 +59,19 @@ Descriptions of section entries and preferred order matches patches or files that contain one or more of the words printk, pr_info or pr_err One regex pattern per line. Multiple K: lines acceptable. - V: *Test suite* proposed for execution. The command that could be - executed to verify changes to the maintained subsystem, or a reference - to a test suite documented in Documentation/process/tests.txt. + V: *Test suite* proposed for execution for verifying changes to the + maintained subsystem. Must start with a proposal strength keyword: + (SUGGESTED/RECOMMENDED/REQUIRED), followed by the test suite command, + or a reference to a test suite documented in + Documentation/process/tests.txt. + Proposal strengths correspond to checkpatch.pl message levels + (CHECK/WARNING/ERROR respectively, whenever Tested-with: is missing). Commands must be executed from the root of the source tree. Commands must support the -h/--help option. References must be preceded with a '*'. Cannot contain '@' or '#' characters. - V: tools/testing/kunit/run_checks.py - V: *xfstests + V: SUGGESTED tools/testing/kunit/run_checks.py + V: RECOMMENDED *xfstests One test suite per line. Maintainers List @@ -7978,7 +7982,7 @@ L: linux-ext4@vger.kernel.org S: Maintained W: http://ext4.wiki.kernel.org Q: http://patchwork.ozlabs.org/project/linux-ext4/list/ -V: *kvm-xfstests smoke +V: RECOMMENDED *kvm-xfstests smoke T: git git://git.kernel.org/pub/scm/linux/kernel/git/tytso/ext4.git F: Documentation/filesystems/ext4/ F: fs/ext4/ @@ -11628,7 +11632,7 @@ L: linux-kselftest@vger.kernel.org L: kunit-dev@googlegroups.com S: Maintained W: https://google.github.io/kunit-docs/third_party/kernel/docs/ -V: tools/testing/kunit/run_checks.py +V: RECOMMENDED tools/testing/kunit/run_checks.py T: git git://git.kernel.org/pub/scm/linux/kernel/git/shuah/linux-kselftest.git kunit T: git git://git.kernel.org/pub/scm/linux/kernel/git/shuah/linux-kselftest.git kunit-fixes F: Documentation/dev-tools/kunit/ @@ -18367,7 +18371,7 @@ REGISTER MAP ABSTRACTION M: Mark Brown L: linux-kernel@vger.kernel.org S: Supported -V: *kunit +V: RECOMMENDED *kunit T: git git://git.kernel.org/pub/scm/linux/kernel/git/broonie/regmap.git F: Documentation/devicetree/bindings/regmap/ F: drivers/base/regmap/ diff --git a/scripts/checkpatch.pl b/scripts/checkpatch.pl index bfeb4c33b5424..9438e4f452a6c 100755 --- a/scripts/checkpatch.pl +++ b/scripts/checkpatch.pl @@ -1181,39 +1181,57 @@ sub is_maintained_obsolete { return $maintained_status{$filename} =~ /obsolete/i; } -# Test suites proposed per changed file +# A list of test proposal strength keywords, weakest to strongest +our @test_proposal_strengths = qw(suggested recommended required); +# A regular expression string matching test proposal strength keywords +my $test_proposal_strengths_res = join('|', @test_proposal_strengths); +# A regular expression matching valid values of MAINTAINERS V: entries +# Puts proposal strength into $1 and the test into $2 +my $test_proposal_entry_re = qr/^\s*($test_proposal_strengths_res)\s+([^@#]+?)\s*$/i; + +# A hashmap of changed files and references to hashmaps of test suites and the +# strength (0-2) of their proposal our %files_proposed_tests = (); -# Return a list of test suites proposed for execution for a particular file +# Return a reference to a hashmap of test suites proposed for execution for a +# particular file, and their proposal strengths - 0, 1, or 2 for +# SUGGESTED/RECOMMENDED/REQUIRED respectively. sub get_file_proposed_tests { my ($filename) = @_; - my $file_proposed_tests; - return () if (!$tree || !(-e "$root/scripts/get_maintainer.pl")); + return {} if (!$tree || !(-e "$root/scripts/get_maintainer.pl")); if (!exists($files_proposed_tests{$filename})) { + # Retrieve and parse the entries + my %file_proposed_tests = (); my $command = "perl $root/scripts/get_maintainer.pl --test --multiline --nogit --nogit-fallback -f $filename"; # Ignore warnings on stderr my $output = `$command 2>/dev/null`; # But regenerate stderr on failure die "Failed retrieving tests proposed for changes to \"$filename\":\n" . `$command 2>&1 >/dev/null` if ($?); - $file_proposed_tests = [grep { !/@/ } split("\n", $output)]; - # Validate and normalize all references - for my $index (0 .. scalar @$file_proposed_tests - 1) { - my $test = $file_proposed_tests->[$index]; - if ($test =~ /^\*\s*(.*)$/) { - my $name = $1; - die "Test $name referenced in MAINTAINERS not found in $testsrelfile\n" - if (!exists $tests{$name}); - $file_proposed_tests->[$index] = "*" . $name; + # For each non-email line (a V: entry) + foreach my $entry (grep { !/@/ } split("\n", $output)) { + # Extract the strength and the test + if ($entry =~ $test_proposal_entry_re) { + my $strength = grep { $test_proposal_strengths[$_] eq lc($1) } + 0..$#test_proposal_strengths; + my $test = $2; + # Validate and normalize references + if ($test =~ /^\*\s*(.*)$/) { + my $name = $1; + die "Test $name referenced in MAINTAINERS not found in $testsrelfile\n" + if (!exists $tests{$name}); + $test = "*" . $name; + } + $file_proposed_tests{$test} = $strength; + } else { + die "Invalid MAINTAINERS V: entry: $entry\n"; } } - $files_proposed_tests{$filename} = $file_proposed_tests; - } else { - $file_proposed_tests = $files_proposed_tests{$filename}; + $files_proposed_tests{$filename} = \%file_proposed_tests; } - return @$file_proposed_tests; + return $files_proposed_tests{$filename}; } sub is_SPDX_License_valid { @@ -2761,7 +2779,7 @@ sub process { my @setup_docs = (); my $setup_docs = 0; - # Test suites which should not be proposed for execution + # Maximum strength (0-2) of test proposals to be ignored my %dont_propose_tests = (); my $camelcase_file_seeded = 0; @@ -2983,8 +3001,10 @@ sub process { } # Check if tests are proposed for changes to the file - foreach my $test (get_file_proposed_tests($realfile)) { - next if exists $dont_propose_tests{$test}; + my $file_proposed_tests = get_file_proposed_tests($realfile); + foreach my $test (keys %$file_proposed_tests) { + my $strength = $file_proposed_tests->{$test}; + next if $strength <= ($dont_propose_tests{$test} // -1); my $name; my $title; my $command; @@ -3000,19 +3020,19 @@ sub process { $command = $test; } if ($command) { - $message = "Execute the $title " . - "proposed for verifying changes to $realfile:\n" . - "$command\n"; + $message = "Execute the $title $test_proposal_strengths[$strength] " . + "for verifying changes to $realfile:\n$command\n"; } else { - $message = "The $title is proposed for verifying changes to $realfile\n"; + $message = "The $title is $test_proposal_strengths[$strength] " . + "for verifying changes to $realfile\n"; } if ($name) { $message .= "See instructions under \"$name\" in $testsrelfile\n"; } $message .= "Add the following to the tested commit's message, " . "IF IT PASSES:\nTested-with: $test\n"; - CHK("TEST_PROPOSAL", $message); - $dont_propose_tests{$test} = 1; + (\&CHK, \&WARN, \&ERROR)[$strength]("TEST_PROPOSAL", $message); + $dont_propose_tests{$test} = $strength; } next; @@ -3350,7 +3370,7 @@ sub process { # the test and its subsets local *dont_propose_test_name = sub { my ($name) = @_; - $dont_propose_tests{"*" . $name} = 1; + $dont_propose_tests{"*" . $name} = 2; foreach my $sub_name (keys %tests) { my $sub_data = $tests{$sub_name}; my $superset = $sub_data->{"superset"}; @@ -3363,7 +3383,7 @@ sub process { # Else it's a command } else { # Do not propose the test - $dont_propose_tests{$test} = 1; + $dont_propose_tests{$test} = 2; } } @@ -3821,9 +3841,10 @@ sub process { if ($rawline =~ /^\+V:\s*(.*)/) { my $entry = $1; # If this is a valid entry value - if ($entry =~ /^[^@#]*$/) { + if ($entry =~ $test_proposal_entry_re) { + my $test = $2; # If the test in the entry is a reference - if ($entry =~ /^\*\s*(.*)$/) { + if ($test =~ /^\*\s*(.*)$/) { my $name = $1; ERROR("TEST_PROPOSAL_INVALID", "Test $name referenced in MAINTAINERS not found in $testsrelfile\n" . @@ -3831,7 +3852,7 @@ sub process { } } else { ERROR("TEST_PROPOSAL_INVALID", - "Test proposal cannot contain '\@' or '#' characters\n" . $herecurr); + "Invalid test proposal entry: $entry\n" . $herecurr); } } }