From patchwork Wed Sep 16 09:42:03 2015 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stuart Haslam X-Patchwork-Id: 53722 Return-Path: X-Original-To: linaro@patches.linaro.org Delivered-To: linaro@patches.linaro.org Received: from mail-lb0-f199.google.com (mail-lb0-f199.google.com [209.85.217.199]) by patches.linaro.org (Postfix) with ESMTPS id 03F9622DB2 for ; Wed, 16 Sep 2015 09:45:21 +0000 (UTC) Received: by lbcao8 with SMTP id ao8sf67140777lbc.1 for ; Wed, 16 Sep 2015 02:45:20 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:delivered-to:delivered-to:from:to:date :message-id:in-reply-to:references:subject:precedence:list-id :list-unsubscribe:list-archive:list-post:list-help:list-subscribe :mime-version:content-type:content-transfer-encoding:errors-to :sender:x-original-sender:x-original-authentication-results :mailing-list; bh=rwcKnD8rbZVtLNtnpyAj7y6EDoLlex5oOGGaNx5Fuy4=; b=NjToxQwG/JwxnvQ9EV/m7Jt9F1/hU+85i4emskEEEkUvIBEsZqz4f/PpaaguvV+/z2 ej+TiJzEYLWNJxbkG6dC3INFoirCIPbHwLSEciv3GqNFLylJamWXrfI2c8ZU3MMK0IN1 JZTAcG2jH62FTvADU+/azWHGfUDynBcxVSKs3YoJRJy1zPTjBGNPGoJb05ThtgaD26Rr GFJVbuSAizxwuoZN5F77wwsJ8A/2f45ulQGg3tvjWLzWX3aJIHxo64KDtVlgLsRP2MQm aLwlShMx8OHZcX1gplingNruPVHhcjDdBLftCYFA0JjyWN1hCMDHp6Dadc5/EdDYI7R+ igAA== X-Gm-Message-State: ALoCoQl75ab+GPFaVWZvPmEsv9m0++jJr4c7pMZjHK7RGY3RbzZbqv9zFRr8aywLTQqPJR1QVCU8 X-Received: by 10.112.63.165 with SMTP id h5mr5351004lbs.18.1442396719991; Wed, 16 Sep 2015 02:45:19 -0700 (PDT) X-BeenThere: patchwork-forward@linaro.org Received: by 10.152.4.98 with SMTP id j2ls13469laj.81.gmail; Wed, 16 Sep 2015 02:45:19 -0700 (PDT) X-Received: by 10.112.24.163 with SMTP id v3mr28016530lbf.101.1442396719836; Wed, 16 Sep 2015 02:45:19 -0700 (PDT) Received: from mail-lb0-f181.google.com (mail-lb0-f181.google.com. [209.85.217.181]) by mx.google.com with ESMTPS id bj8si17638417lbc.93.2015.09.16.02.45.19 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 16 Sep 2015 02:45:19 -0700 (PDT) Received-SPF: pass (google.com: domain of patch+caf_=patchwork-forward=linaro.org@linaro.org designates 209.85.217.181 as permitted sender) client-ip=209.85.217.181; Received: by lbcao8 with SMTP id ao8so100032042lbc.3 for ; Wed, 16 Sep 2015 02:45:19 -0700 (PDT) X-Received: by 10.112.146.104 with SMTP id tb8mr26102357lbb.35.1442396719593; Wed, 16 Sep 2015 02:45:19 -0700 (PDT) X-Forwarded-To: patchwork-forward@linaro.org X-Forwarded-For: patch@linaro.org patchwork-forward@linaro.org Delivered-To: patch@linaro.org Received: by 10.112.59.35 with SMTP id w3csp2328412lbq; Wed, 16 Sep 2015 02:45:18 -0700 (PDT) X-Received: by 10.55.195.86 with SMTP id a83mr39357354qkj.72.1442396718583; Wed, 16 Sep 2015 02:45:18 -0700 (PDT) Received: from lists.linaro.org (lists.linaro.org. [54.225.227.206]) by mx.google.com with ESMTP id l96si21157129qgl.119.2015.09.16.02.45.16; Wed, 16 Sep 2015 02:45:18 -0700 (PDT) Received-SPF: pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.225.227.206 as permitted sender) client-ip=54.225.227.206; Received: by lists.linaro.org (Postfix, from userid 109) id B714861A34; Wed, 16 Sep 2015 09:45:16 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on ip-10-142-244-252 X-Spam-Level: X-Spam-Status: No, score=-2.6 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_LOW, RCVD_IN_MSPIKE_H3, RCVD_IN_MSPIKE_WL, URIBL_BLOCKED autolearn=disabled version=3.4.0 Received: from [127.0.0.1] (localhost [127.0.0.1]) by lists.linaro.org (Postfix) with ESMTP id 8201361FCC; Wed, 16 Sep 2015 09:43:59 +0000 (UTC) X-Original-To: lng-odp@lists.linaro.org Delivered-To: lng-odp@lists.linaro.org Received: by lists.linaro.org (Postfix, from userid 109) id AA202619BA; Wed, 16 Sep 2015 09:43:50 +0000 (UTC) Received: from mail-wi0-f177.google.com (mail-wi0-f177.google.com [209.85.212.177]) by lists.linaro.org (Postfix) with ESMTPS id 6651261A29 for ; Wed, 16 Sep 2015 09:42:42 +0000 (UTC) Received: by wiclk2 with SMTP id lk2so64460053wic.0 for ; Wed, 16 Sep 2015 02:42:41 -0700 (PDT) X-Received: by 10.180.106.4 with SMTP id gq4mr17357109wib.42.1442396561554; Wed, 16 Sep 2015 02:42:41 -0700 (PDT) Received: from e106441.emea.arm.com ([2001:41d0:a:3cb4::abcd]) by smtp.gmail.com with ESMTPSA id kb5sm25580683wjc.17.2015.09.16.02.42.40 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Wed, 16 Sep 2015 02:42:40 -0700 (PDT) From: Stuart Haslam To: lng-odp@lists.linaro.org Date: Wed, 16 Sep 2015 10:42:03 +0100 Message-Id: <1442396523-18466-5-git-send-email-stuart.haslam@linaro.org> X-Mailer: git-send-email 2.1.1 In-Reply-To: <1442396523-18466-1-git-send-email-stuart.haslam@linaro.org> References: <1442396523-18466-1-git-send-email-stuart.haslam@linaro.org> X-Topics: patch Subject: [lng-odp] [PATCH 4/4] doc: implementers-guide: update section on skipping tests X-BeenThere: lng-odp@lists.linaro.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: , List-Help: , List-Subscribe: , MIME-Version: 1.0 Errors-To: lng-odp-bounces@lists.linaro.org Sender: "lng-odp" X-Removed-Original-Auth: Dkim didn't pass. X-Original-Sender: stuart.haslam@linaro.org X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of patch+caf_=patchwork-forward=linaro.org@linaro.org designates 209.85.217.181 as permitted sender) smtp.mailfrom=patch+caf_=patchwork-forward=linaro.org@linaro.org Mailing-list: list patchwork-forward@linaro.org; contact patchwork-forward+owners@linaro.org X-Google-Group-Id: 836684582541 Update sections describing how a specific platform may skip tests by marking them as inactive. Signed-off-by: Stuart Haslam --- doc/implementers-guide/implementers-guide.adoc | 58 +++++++++++++++++++++----- 1 file changed, 48 insertions(+), 10 deletions(-) diff --git a/doc/implementers-guide/implementers-guide.adoc b/doc/implementers-guide/implementers-guide.adoc index 090d4e5..181d5c2 100644 --- a/doc/implementers-guide/implementers-guide.adoc +++ b/doc/implementers-guide/implementers-guide.adoc @@ -110,7 +110,7 @@ Module test and naming convention *_test_+++*+++* + where the suffix idendify the test. -* Test arrays, i.e. arrays of CU_TestInfo, listing the test functions belonging to a suite, are called: +* Test arrays, i.e. arrays of odp_testinfo_t, listing the test functions belonging to a suite, are called: *_suite+++[_*]+++* + where the possible suffix can be used if many suites are declared. @@ -118,7 +118,7 @@ Module test and naming convention *+++_suite[_*]_init()+++* and *+++_suite[_*]_term()+++* respectively. + where the possible extra middle pattern can be used if many suites are declared. -* Suite arrays, i.e. arrays of CU_SuiteInfo used in executables (C_UNIT registry) are called: +* Suite arrays, i.e. arrays of odp_suiteinfo_t used in executables (C_UNIT registry) are called: *+++_suites[_*]+++* + where the possible suffix identifies the executable using it, if many. @@ -232,14 +232,52 @@ Defining platform specific tests Sometimes, it may be necessary to call platform specific system calls to check some functionality: For instance, testing odp_cpumask_* could involve checking the underlying system CPU mask. On linux, such a test would require using the CPU_ISSET macro, which is linux specific. Such a test would be written in '/cpumask/...' The contents of this directory would be very similar to the contents of the platform agnostic side cpu_mask tests (including a Makefile.am...), but platform specific test would be written there. '/Makefile.am' would then trigger the building of the platform specific tests (by listing their module name in SUBDIRS and therefore calling the appropriate Makefile.am) and then it would call both the platform agnostic executable(s) and the platform specific test executable. -Skipping tests during development -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -During ODP development, it may be useful to skip some test. This can be achieved by creating a new test executable (still on the platform side), picking up the required tests from the platform agnostic libtest.la. +Marking tests as inactive +^^^^^^^^^^^^^^^^^^^^^^^^^ +The general policy is that a full run of the validation suite (a "make check") must pass at all times. However a particular platform may have one or more test cases that are known to be unimplemented either during development or permanently, so to avoid these test cases being reported as failures it's useful to be able to skip them. This can be achieved by creating a new test executable (still on the platform side), giving the platform specific initialisation code the opportunity to modify the registered tests in order to mark unwanted tests as inactive while leaving the remaining tests active. It's important that the unwanted tests are still registered with the test framework to allow the fact that they're not being tested to be recorded. + +The odp_cunit_update() function is intended for this purpose, it is used to modify the properties of previously registered tests, for example to mark them as inactive. Inactive tests are registered with the test framework but aren't executed and will be recorded as inactive in test reports. + +In 'test/validation/foo/foo.c', define all tests for the 'foo' module; + +[source,c] +------------------ +odp_testinfo_t foo_tests[] = { + ODP_TEST_INFO(foo_test_a), + ODP_TEST_INFO(foo_test_b), + ODP_TEST_INFO_NULL +}; + +odp_suiteinfo_t foo_suites[] = { + {"Foo", foo_suite_init, foo_suite_term, foo_tests}, + ODP_SUITE_INFO_NULL +}; +------------------ + +In 'platform//test/foo/foo_main.c', register all the tests defined in the 'foo' module, then mark a single specific test case as inactive; + +[source,c] +------------------ +static odp_testinfo_t foo_tests_updates[] = { + ODP_TEST_INFO_INACTIVE(foo_test_b), + ODP_TEST_INFO_NULL +}; + +static odp_suiteinfo_t foo_suites_updates[] = { + {"Foo", foo_suite_init, foo_suite_term, foo_tests_updates}, + ODP_SUITE_INFO_NULL +}; + +int pktio_main(void) +{ + int ret = odp_cunit_register(foo_suites); -The top Makefile would then call only the platform specific executable, hence skipping the tests which have been omitted. + if (ret == 0) + ret = odp_cuint_update(foo_suites_updates); -TIP: You probably want to copy the platform-agnostic module main function and prune it from the undesired tests when writing your own platform specific main, for a given module. + if (ret == 0) + ret = odp_cunit_run(); -Permanently skipping test -^^^^^^^^^^^^^^^^^^^^^^^^^^ -If a platform wants to permanently skip a test (i.e. a part of the ODP API is and will not be supported on that platform), it is recommended to use the function odp_cunit_TBD() to removed the tests or suite from the list of tests. This gives a chance to the test environment to trace this removal. + return ret; +} +------------------