| Message ID | 1442396523-18466-5-git-send-email-stuart.haslam@linaro.org |
|---|---|
| State | New |
| Headers | show |
On 2015-09-16 10:42, Stuart Haslam wrote: > Update sections describing how a specific platform may skip tests by > marking them as inactive. > > Signed-off-by: Stuart Haslam <stuart.haslam@linaro.org> > --- > doc/implementers-guide/implementers-guide.adoc | 58 +++++++++++++++++++++----- > 1 file changed, 48 insertions(+), 10 deletions(-) > > diff --git a/doc/implementers-guide/implementers-guide.adoc b/doc/implementers-guide/implementers-guide.adoc > index 090d4e5..181d5c2 100644 > --- a/doc/implementers-guide/implementers-guide.adoc > +++ b/doc/implementers-guide/implementers-guide.adoc > @@ -110,7 +110,7 @@ Module test and naming convention > *<Module>_test_+++*+++* + > where the suffix idendify the test. > > -* Test arrays, i.e. arrays of CU_TestInfo, listing the test functions belonging to a suite, are called: > +* Test arrays, i.e. arrays of odp_testinfo_t, listing the test functions belonging to a suite, are called: > *<Module>_suite+++[_*]+++* + > where the possible suffix can be used if many suites are declared. > > @@ -118,7 +118,7 @@ Module test and naming convention > *<Module>+++_suite[_*]_init()+++* and *<Module>+++_suite[_*]_term()+++* respectively. + > where the possible extra middle pattern can be used if many suites are declared. > > -* Suite arrays, i.e. arrays of CU_SuiteInfo used in executables (C_UNIT registry) are called: > +* Suite arrays, i.e. arrays of odp_suiteinfo_t used in executables (C_UNIT registry) are called: > *<Module>+++_suites[_*]+++* + > where the possible suffix identifies the executable using it, if many. > > @@ -232,14 +232,52 @@ Defining platform specific tests > Sometimes, it may be necessary to call platform specific system calls to check some functionality: For instance, testing odp_cpumask_* could involve checking the underlying system CPU mask. On linux, such a test would require using the CPU_ISSET macro, which is linux specific. Such a test would be written in '<PLATFORM_SPECIFIC>/cpumask/...' The contents of this directory would be very similar to the contents of the platform agnostic side cpu_mask tests (including a Makefile.am...), but platform specific test would be written there. > '<PLATFORM_SPECIFIC>/Makefile.am' would then trigger the building of the platform specific tests (by listing their module name in SUBDIRS and therefore calling the appropriate Makefile.am) and then it would call both the platform agnostic executable(s) and the platform specific test executable. > > -Skipping tests during development > -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > -During ODP development, it may be useful to skip some test. This can be achieved by creating a new test executable (still on the platform side), picking up the required tests from the platform agnostic libtest<module>.la. > +Marking tests as inactive > +^^^^^^^^^^^^^^^^^^^^^^^^^ > +The general policy is that a full run of the validation suite (a "make check") must pass at all times. However a particular platform may have one or more test cases that are known to be unimplemented either during development or permanently, so to avoid these test cases being reported as failures it's useful to be able to skip them. This can be achieved by creating a new test executable (still on the platform side), giving the platform specific initialisation code the opportunity to modify the registered tests in order to mark unwanted tests as inactive while leaving the remaining tests active. It's important that the unwanted tests are still registered with the test framework to allow the fact that they're not being tested to be recorded. > + > +The odp_cunit_update() function is intended for this purpose, it is used to modify the properties of previously registered tests, for example to mark them as inactive. Inactive tests are registered with the test framework but aren't executed and will be recorded as inactive in test reports. > + > +In 'test/validation/foo/foo.c', define all tests for the 'foo' module; > + > +[source,c] > +------------------ > +odp_testinfo_t foo_tests[] = { > + ODP_TEST_INFO(foo_test_a), > + ODP_TEST_INFO(foo_test_b), > + ODP_TEST_INFO_NULL > +}; > + > +odp_suiteinfo_t foo_suites[] = { > + {"Foo", foo_suite_init, foo_suite_term, foo_tests}, > + ODP_SUITE_INFO_NULL > +}; > +------------------ > + > +In 'platform/<platform>/test/foo/foo_main.c', register all the tests defined in the 'foo' module, then mark a single specific test case as inactive; > + > +[source,c] > +------------------ > +static odp_testinfo_t foo_tests_updates[] = { > + ODP_TEST_INFO_INACTIVE(foo_test_b), > + ODP_TEST_INFO_NULL > +}; > + > +static odp_suiteinfo_t foo_suites_updates[] = { > + {"Foo", foo_suite_init, foo_suite_term, foo_tests_updates}, > + ODP_SUITE_INFO_NULL > +}; > + > +int pktio_main(void) should be foo_main > +{ > + int ret = odp_cunit_register(foo_suites); > > -The top Makefile would then call only the platform specific executable, hence skipping the tests which have been omitted. > + if (ret == 0) > + ret = odp_cuint_update(foo_suites_updates); > > -TIP: You probably want to copy the platform-agnostic module main function and prune it from the undesired tests when writing your own platform specific main, for a given module. > + if (ret == 0) > + ret = odp_cunit_run(); > > -Permanently skipping test > -^^^^^^^^^^^^^^^^^^^^^^^^^^ > -If a platform wants to permanently skip a test (i.e. a part of the ODP API is and will not be supported on that platform), it is recommended to use the function odp_cunit_TBD() to removed the tests or suite from the list of tests. This gives a chance to the test environment to trace this removal. > + return ret; > +} > +------------------ Maybe it should say that is is possible, during devloppement, to skip all validation tests and have only platform specific if needed, and then beeing able to add the common tests one by one. I know we don't exactely have the same view here... but we are getting closer :-) Thanks anyway, we are clearly making progress! /Christophe > -- > 2.1.1 > > _______________________________________________ > lng-odp mailing list > lng-odp@lists.linaro.org > https://lists.linaro.org/mailman/listinfo/lng-odp
On Wed, Sep 16, 2015 at 05:19:28PM +0200, Christophe Milard wrote: > On 2015-09-16 10:42, Stuart Haslam wrote: > > Update sections describing how a specific platform may skip tests by > > marking them as inactive. > > > > Signed-off-by: Stuart Haslam <stuart.haslam@linaro.org> > > --- > > doc/implementers-guide/implementers-guide.adoc | 58 +++++++++++++++++++++----- > > 1 file changed, 48 insertions(+), 10 deletions(-) > > > > diff --git a/doc/implementers-guide/implementers-guide.adoc b/doc/implementers-guide/implementers-guide.adoc > > index 090d4e5..181d5c2 100644 > > --- a/doc/implementers-guide/implementers-guide.adoc > > +++ b/doc/implementers-guide/implementers-guide.adoc > > @@ -110,7 +110,7 @@ Module test and naming convention > > *<Module>_test_+++*+++* + > > where the suffix idendify the test. > > > > -* Test arrays, i.e. arrays of CU_TestInfo, listing the test functions belonging to a suite, are called: > > +* Test arrays, i.e. arrays of odp_testinfo_t, listing the test functions belonging to a suite, are called: > > *<Module>_suite+++[_*]+++* + > > where the possible suffix can be used if many suites are declared. > > > > @@ -118,7 +118,7 @@ Module test and naming convention > > *<Module>+++_suite[_*]_init()+++* and *<Module>+++_suite[_*]_term()+++* respectively. + > > where the possible extra middle pattern can be used if many suites are declared. > > > > -* Suite arrays, i.e. arrays of CU_SuiteInfo used in executables (C_UNIT registry) are called: > > +* Suite arrays, i.e. arrays of odp_suiteinfo_t used in executables (C_UNIT registry) are called: > > *<Module>+++_suites[_*]+++* + > > where the possible suffix identifies the executable using it, if many. > > > > @@ -232,14 +232,52 @@ Defining platform specific tests > > Sometimes, it may be necessary to call platform specific system calls to check some functionality: For instance, testing odp_cpumask_* could involve checking the underlying system CPU mask. On linux, such a test would require using the CPU_ISSET macro, which is linux specific. Such a test would be written in '<PLATFORM_SPECIFIC>/cpumask/...' The contents of this directory would be very similar to the contents of the platform agnostic side cpu_mask tests (including a Makefile.am...), but platform specific test would be written there. > > '<PLATFORM_SPECIFIC>/Makefile.am' would then trigger the building of the platform specific tests (by listing their module name in SUBDIRS and therefore calling the appropriate Makefile.am) and then it would call both the platform agnostic executable(s) and the platform specific test executable. > > > > -Skipping tests during development > > -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > > -During ODP development, it may be useful to skip some test. This can be achieved by creating a new test executable (still on the platform side), picking up the required tests from the platform agnostic libtest<module>.la. > > +Marking tests as inactive > > +^^^^^^^^^^^^^^^^^^^^^^^^^ > > +The general policy is that a full run of the validation suite (a "make check") must pass at all times. However a particular platform may have one or more test cases that are known to be unimplemented either during development or permanently, so to avoid these test cases being reported as failures it's useful to be able to skip them. This can be achieved by creating a new test executable (still on the platform side), giving the platform specific initialisation code the opportunity to modify the registered tests in order to mark unwanted tests as inactive while leaving the remaining tests active. It's important that the unwanted tests are still registered with the test framework to allow the fact that they're not being tested to be recorded. > > + > > +The odp_cunit_update() function is intended for this purpose, it is used to modify the properties of previously registered tests, for example to mark them as inactive. Inactive tests are registered with the test framework but aren't executed and will be recorded as inactive in test reports. > > + > > +In 'test/validation/foo/foo.c', define all tests for the 'foo' module; > > + > > +[source,c] > > +------------------ > > +odp_testinfo_t foo_tests[] = { > > + ODP_TEST_INFO(foo_test_a), > > + ODP_TEST_INFO(foo_test_b), > > + ODP_TEST_INFO_NULL > > +}; > > + > > +odp_suiteinfo_t foo_suites[] = { > > + {"Foo", foo_suite_init, foo_suite_term, foo_tests}, > > + ODP_SUITE_INFO_NULL > > +}; > > +------------------ > > + > > +In 'platform/<platform>/test/foo/foo_main.c', register all the tests defined in the 'foo' module, then mark a single specific test case as inactive; > > + > > +[source,c] > > +------------------ > > +static odp_testinfo_t foo_tests_updates[] = { > > + ODP_TEST_INFO_INACTIVE(foo_test_b), > > + ODP_TEST_INFO_NULL > > +}; > > + > > +static odp_suiteinfo_t foo_suites_updates[] = { > > + {"Foo", foo_suite_init, foo_suite_term, foo_tests_updates}, > > + ODP_SUITE_INFO_NULL > > +}; > > + > > +int pktio_main(void) > > should be foo_main > > > +{ > > + int ret = odp_cunit_register(foo_suites); > > > > -The top Makefile would then call only the platform specific executable, hence skipping the tests which have been omitted. > > + if (ret == 0) > > + ret = odp_cuint_update(foo_suites_updates); > > > > -TIP: You probably want to copy the platform-agnostic module main function and prune it from the undesired tests when writing your own platform specific main, for a given module. > > + if (ret == 0) > > + ret = odp_cunit_run(); > > > > -Permanently skipping test > > -^^^^^^^^^^^^^^^^^^^^^^^^^^ > > -If a platform wants to permanently skip a test (i.e. a part of the ODP API is and will not be supported on that platform), it is recommended to use the function odp_cunit_TBD() to removed the tests or suite from the list of tests. This gives a chance to the test environment to trace this removal. > > + return ret; > > +} > > +------------------ > > Maybe it should say that is is possible, during devloppement, to skip all validation tests and have only platform specific if needed, and then beeing able to add the common tests one by one. OK I'll expand it. > I know we don't exactely have the same view here... but we are getting closer :-) Actually this bit I agree with. Selectively running tests is fine, it's swapping them out that I'm not convinced about (but I'm happy enough to ignore it until it actually causes a problem).
diff --git a/doc/implementers-guide/implementers-guide.adoc b/doc/implementers-guide/implementers-guide.adoc index 090d4e5..181d5c2 100644 --- a/doc/implementers-guide/implementers-guide.adoc +++ b/doc/implementers-guide/implementers-guide.adoc @@ -110,7 +110,7 @@ Module test and naming convention *<Module>_test_+++*+++* + where the suffix idendify the test. -* Test arrays, i.e. arrays of CU_TestInfo, listing the test functions belonging to a suite, are called: +* Test arrays, i.e. arrays of odp_testinfo_t, listing the test functions belonging to a suite, are called: *<Module>_suite+++[_*]+++* + where the possible suffix can be used if many suites are declared. @@ -118,7 +118,7 @@ Module test and naming convention *<Module>+++_suite[_*]_init()+++* and *<Module>+++_suite[_*]_term()+++* respectively. + where the possible extra middle pattern can be used if many suites are declared. -* Suite arrays, i.e. arrays of CU_SuiteInfo used in executables (C_UNIT registry) are called: +* Suite arrays, i.e. arrays of odp_suiteinfo_t used in executables (C_UNIT registry) are called: *<Module>+++_suites[_*]+++* + where the possible suffix identifies the executable using it, if many. @@ -232,14 +232,52 @@ Defining platform specific tests Sometimes, it may be necessary to call platform specific system calls to check some functionality: For instance, testing odp_cpumask_* could involve checking the underlying system CPU mask. On linux, such a test would require using the CPU_ISSET macro, which is linux specific. Such a test would be written in '<PLATFORM_SPECIFIC>/cpumask/...' The contents of this directory would be very similar to the contents of the platform agnostic side cpu_mask tests (including a Makefile.am...), but platform specific test would be written there. '<PLATFORM_SPECIFIC>/Makefile.am' would then trigger the building of the platform specific tests (by listing their module name in SUBDIRS and therefore calling the appropriate Makefile.am) and then it would call both the platform agnostic executable(s) and the platform specific test executable. -Skipping tests during development -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -During ODP development, it may be useful to skip some test. This can be achieved by creating a new test executable (still on the platform side), picking up the required tests from the platform agnostic libtest<module>.la. +Marking tests as inactive +^^^^^^^^^^^^^^^^^^^^^^^^^ +The general policy is that a full run of the validation suite (a "make check") must pass at all times. However a particular platform may have one or more test cases that are known to be unimplemented either during development or permanently, so to avoid these test cases being reported as failures it's useful to be able to skip them. This can be achieved by creating a new test executable (still on the platform side), giving the platform specific initialisation code the opportunity to modify the registered tests in order to mark unwanted tests as inactive while leaving the remaining tests active. It's important that the unwanted tests are still registered with the test framework to allow the fact that they're not being tested to be recorded. + +The odp_cunit_update() function is intended for this purpose, it is used to modify the properties of previously registered tests, for example to mark them as inactive. Inactive tests are registered with the test framework but aren't executed and will be recorded as inactive in test reports. + +In 'test/validation/foo/foo.c', define all tests for the 'foo' module; + +[source,c] +------------------ +odp_testinfo_t foo_tests[] = { + ODP_TEST_INFO(foo_test_a), + ODP_TEST_INFO(foo_test_b), + ODP_TEST_INFO_NULL +}; + +odp_suiteinfo_t foo_suites[] = { + {"Foo", foo_suite_init, foo_suite_term, foo_tests}, + ODP_SUITE_INFO_NULL +}; +------------------ + +In 'platform/<platform>/test/foo/foo_main.c', register all the tests defined in the 'foo' module, then mark a single specific test case as inactive; + +[source,c] +------------------ +static odp_testinfo_t foo_tests_updates[] = { + ODP_TEST_INFO_INACTIVE(foo_test_b), + ODP_TEST_INFO_NULL +}; + +static odp_suiteinfo_t foo_suites_updates[] = { + {"Foo", foo_suite_init, foo_suite_term, foo_tests_updates}, + ODP_SUITE_INFO_NULL +}; + +int pktio_main(void) +{ + int ret = odp_cunit_register(foo_suites); -The top Makefile would then call only the platform specific executable, hence skipping the tests which have been omitted. + if (ret == 0) + ret = odp_cuint_update(foo_suites_updates); -TIP: You probably want to copy the platform-agnostic module main function and prune it from the undesired tests when writing your own platform specific main, for a given module. + if (ret == 0) + ret = odp_cunit_run(); -Permanently skipping test -^^^^^^^^^^^^^^^^^^^^^^^^^^ -If a platform wants to permanently skip a test (i.e. a part of the ODP API is and will not be supported on that platform), it is recommended to use the function odp_cunit_TBD() to removed the tests or suite from the list of tests. This gives a chance to the test environment to trace this removal. + return ret; +} +------------------
Update sections describing how a specific platform may skip tests by marking them as inactive. Signed-off-by: Stuart Haslam <stuart.haslam@linaro.org> --- doc/implementers-guide/implementers-guide.adoc | 58 +++++++++++++++++++++----- 1 file changed, 48 insertions(+), 10 deletions(-)