From patchwork Wed Jul 13 16:08:23 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christophe Milard X-Patchwork-Id: 71938 Delivered-To: patch@linaro.org Received: by 10.140.29.52 with SMTP id a49csp1085596qga; Wed, 13 Jul 2016 09:12:36 -0700 (PDT) X-Received: by 10.55.42.78 with SMTP id q75mr11570241qkh.158.1468426356382; Wed, 13 Jul 2016 09:12:36 -0700 (PDT) Return-Path: Received: from lists.linaro.org (lists.linaro.org. [54.225.227.206]) by mx.google.com with ESMTP id b34si1056518qtb.136.2016.07.13.09.12.36; Wed, 13 Jul 2016 09:12:36 -0700 (PDT) Received-SPF: pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.225.227.206 as permitted sender) client-ip=54.225.227.206; Authentication-Results: mx.google.com; spf=pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.225.227.206 as permitted sender) smtp.mailfrom=lng-odp-bounces@lists.linaro.org; dmarc=pass (p=NONE dis=NONE) header.from=linaro.org Received: by lists.linaro.org (Postfix, from userid 109) id 0F8AF62EDB; Wed, 13 Jul 2016 16:12:36 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on ip-10-142-244-252 X-Spam-Level: X-Spam-Status: No, score=-2.6 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_LOW, URIBL_BLOCKED autolearn=disabled version=3.4.0 Received: from [127.0.0.1] (localhost [127.0.0.1]) by lists.linaro.org (Postfix) with ESMTP id CE82C685FF; Wed, 13 Jul 2016 16:09:06 +0000 (UTC) X-Original-To: lng-odp@lists.linaro.org Delivered-To: lng-odp@lists.linaro.org Received: by lists.linaro.org (Postfix, from userid 109) id 9A2D7685B1; Wed, 13 Jul 2016 16:08:43 +0000 (UTC) Received: from mail-lf0-f51.google.com (mail-lf0-f51.google.com [209.85.215.51]) by lists.linaro.org (Postfix) with ESMTPS id 220BB6806E for ; Wed, 13 Jul 2016 16:08:38 +0000 (UTC) Received: by mail-lf0-f51.google.com with SMTP id b199so43100679lfe.0 for ; Wed, 13 Jul 2016 09:08:38 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=uUXUVM5HUs1s9/CARpP0ddUcqGz/BS0N9XQ2TW5jyfk=; b=BknVpT2UyJrybvnJwY9bt/EfPI24qnkdtu1DZMZlsIozX8soRGBfq45hvrhPj9PAx/ eO0qr9Tccu8MWDJm9xVdfpRFVGVxS4PONrIICR2QRjD00PIb7jUuNjR0hWWenOfhCV22 vlwYbyoEOFnGKOxwxax01n1j/ppfx94qiZ1rton+N6QtBUcdwVYRL96EU34JvQ/yyxRV yTjJXIjjQB/0D/ktK58JoVioAdfmRw9YlGM6V3GOhq+EtSo6+d+PWOwa6Z9gxnHgUey1 1/Z3we0kTPOJ2L3mt173xPQMvcrDwgxq0N9N3SsulNKumq7CbPnhpiEgnf87F4pEAVXb YUeg== X-Gm-Message-State: ALyK8tIihelwbuXqW7dD7KBi/+3AcQf9W9wRsiBfC/DnBESmFKVmpWXQ0KQzGYbSXzqj7NcPtoM= X-Received: by 10.25.156.77 with SMTP id f74mr246007lfe.51.1468426116797; Wed, 13 Jul 2016 09:08:36 -0700 (PDT) Received: from localhost.localdomain (c-83-233-90-46.cust.bredband2.com. [83.233.90.46]) by smtp.gmail.com with ESMTPSA id 88sm290103lfr.23.2016.07.13.09.08.35 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Wed, 13 Jul 2016 09:08:35 -0700 (PDT) From: Christophe Milard To: mike.holmes@linaro.org, balakrishna.garapati@linaro.org, yi.he@linaro.org, lng-odp@lists.linaro.org, bill.fischofer@linaro.org Date: Wed, 13 Jul 2016 18:08:23 +0200 Message-Id: <1468426103-20168-5-git-send-email-christophe.milard@linaro.org> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1468426103-20168-1-git-send-email-christophe.milard@linaro.org> References: <1468426103-20168-1-git-send-email-christophe.milard@linaro.org> X-Topics: patch Subject: [lng-odp] [PATCHv2 4/4] doc: implementers guide update with the new structure X-BeenThere: lng-odp@lists.linaro.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: "The OpenDataPlane \(ODP\) List" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: lng-odp-bounces@lists.linaro.org Sender: "lng-odp" Signed-off-by: Christophe Milard --- doc/implementers-guide/implementers-guide.adoc | 137 +++++++++++++++---------- 1 file changed, 81 insertions(+), 56 deletions(-) -- 2.7.4 diff --git a/doc/implementers-guide/implementers-guide.adoc b/doc/implementers-guide/implementers-guide.adoc index 4f49e8f..cb09f38 100644 --- a/doc/implementers-guide/implementers-guide.adoc +++ b/doc/implementers-guide/implementers-guide.adoc @@ -227,21 +227,30 @@ the underlying platform. They are written in plain C code, and may only use functions defined in the standard libC (C99) library (besides the ODP functions being tested, of course). A free C99 specification can be found at the http://www.open-std.org/JTC1/sc22/wg14/www/docs/n1256.pdf[open-std.org] -web site. No other languages (like scripting) are allowed as their usage +web site. No other languages (like scripting) are allowed as their usage would make assumptions on the platform capability. -This area is located at `test/validation/` in the odp git repository. +This area is located at: `test/common_plat/` -The ODP API itself is ordered by module, where each module groups the set of -ODP API functions related to the same "topic". Examples of modules includes -"classification" (API functions dealing with ingress packets classification), -time (functions dealing with time, excluding timers which have their own -module), timer,... The complete module list can be seen at: -http://docs.opendataplane.org/master/linux-generic-doxygen-html/modules.html[ODP -Modules] + Within the platform agnostic area, the tests are also grouped by -modules, matching the ODP API modules: `test/validation/` mainly contains a -list of directories matching each module name (as defined by the doxygen -`@defgroup` or `@ingroup` statement present in each API `.h` file). +In this directory, tests are grouped by category: + +* validation : groups of test defining the ODP compliance +* performance : tests checking system responsiveness +* miscellaneous + +Each ODP interface contains modules, where each module groups the set of ODP +functions related to the same "topic" for the given interface. +Examples of modules for the application interface includes "classification" +(API functions dealing with ingress packets classification), time +(functions dealing with time, excluding timers which have their own module), +timer, ... +The complete module list can be seen at: +http://docs.opendataplane.org/master/linux-generic-doxygen-html/modules.html[ODP Modules] + +Within the platform agnostic area, the validation tests for a given interface +are also grouped by modules, matching the ODP interface modules: for instance, +`test/common_plat/validation/api` mainly contains a list of +directories matching each module name (as defined by the doxygen `@defgroup` +or `@ingroup` statement present in each API `.h` file). Within each of these directories, a library (called `libtest.la`) and its associated `.h` file (called `.h`) defines all the test functions @@ -253,9 +262,10 @@ for this module. See <> for more details. It is important to be aware that the tests defined for a given module -(defined in `test/validation/`) are focused to test the ODP functions -belonging to this module, but are not limited to use this module's ODP functions -only: many modules needs some interaction with some other module to be tested. +(defined in `test/common_plat/validation/api/`) are focused to test +the ODP functions belonging to this module, but are not limited to use this +module's ODP functions only: many modules needs some interaction with some +other module to be tested. The obvious illustration of this is for module "init" whose functions are required by all tests of all other modules (as ODP needs to be initialized to test anything else). @@ -317,8 +327,8 @@ library. === Platform specific -These tests are located under `platform//test`. There is one such -area for each platform implementing ODP. This location will be referred as +These tests are located under `platform//`. There is one such +area for each platform implementing ODP. This location will be referred as in the rest of this document. ==== The normal case @@ -329,55 +339,64 @@ simply needs to contain a single `Makefile.am` listing each of the executables listed in the automake TEST variable and will therefore be run on "make check". For the linux-generic platform, most tested modules fall into this category: -currently, the `platform/linux-generic/test/Makefile.am` looks as follows: +currently, the `test/linux-generic/test/Makefile.am` looks +roughly as follows: [source,am] ---- include $(top_srcdir)/test/Makefile.inc -TESTS_ENVIRONMENT += TEST_DIR=${top_builddir}/test/validation +TESTS_ENVIRONMENT += TEST_DIR=${top_builddir}/test/common_plat/validation + +ALL_API_VALIDATION = ${top_builddir}/test/common_plat/validation/api -ODP_MODULES = pktio +SUBDIRS = if test_vald -TESTS = pktio/pktio_run \ - ${top_builddir}/test/validation/buffer/buffer_main$(EXEEXT) \ - ${top_builddir}/test/validation/classification/classification_main$(EXEEXT) \ - ${top_builddir}/test/validation/cpumask/cpumask_main$(EXEEXT) \ - ${top_builddir}/test/validation/crypto/crypto_main$(EXEEXT) \ - ${top_builddir}/test/validation/errno/errno_main$(EXEEXT) \ - ${top_builddir}/test/validation/init/init_main_ok$(EXEEXT) \ - ${top_builddir}/test/validation/init/init_main_abort$(EXEEXT) \ - ${top_builddir}/test/validation/init/init_main_log$(EXEEXT) \ - ${top_builddir}/test/validation/packet/packet_main$(EXEEXT) \ - ${top_builddir}/test/validation/pool/pool_main$(EXEEXT) \ - ${top_builddir}/test/validation/queue/queue_main$(EXEEXT) \ - ${top_builddir}/test/validation/random/random_main$(EXEEXT) \ - ${top_builddir}/test/validation/scheduler/scheduler_main$(EXEEXT) \ - ${top_builddir}/test/validation/synchronizers/synchronizers_main$(EXEEXT) \ - ${top_builddir}/test/validation/thread/thread_main$(EXEEXT) \ - ${top_builddir}/test/validation/time/time_main$(EXEEXT) \ - ${top_builddir}/test/validation/timer/timer_main$(EXEEXT) \ - ${top_builddir}/test/validation/shmem/shmem_main$(EXEEXT) \ - ${top_builddir}/test/validation/system/system_main$(EXEEXT) - -SUBDIRS = $(ODP_MODULES) -endif +TESTS = validation/api/pktio/pktio_run.sh \ + validation/api/pktio/pktio_run_tap.sh \ + $(ALL_API_VALIDATION)/atomic/atomic_main$(EXEEXT) \ + $(ALL_API_VALIDATION)/barrier/barrier_main$(EXEEXT) \ + $(ALL_API_VALIDATION)/buffer/buffer_main$(EXEEXT) \ + $(ALL_API_VALIDATION)/classification/classification_main$(EXEEXT) \ + $(ALL_API_VALIDATION)/cpumask/cpumask_main$(EXEEXT) \ + $(ALL_API_VALIDATION)/crypto/crypto_main$(EXEEXT) \ + $(ALL_API_VALIDATION)/errno/errno_main$(EXEEXT) \ + $(ALL_API_VALIDATION)/hash/hash_main$(EXEEXT) \ + $(ALL_API_VALIDATION)/init/init_main_ok$(EXEEXT) \ + $(ALL_API_VALIDATION)/init/init_main_abort$(EXEEXT) \ + $(ALL_API_VALIDATION)/init/init_main_log$(EXEEXT) \ + $(ALL_API_VALIDATION)/lock/lock_main$(EXEEXT) \ + $(ALL_API_VALIDATION)/packet/packet_main$(EXEEXT) \ + $(ALL_API_VALIDATION)/pool/pool_main$(EXEEXT) \ + $(ALL_API_VALIDATION)/queue/queue_main$(EXEEXT) \ + $(ALL_API_VALIDATION)/random/random_main$(EXEEXT) \ + $(ALL_API_VALIDATION)/scheduler/scheduler_main$(EXEEXT) \ + $(ALL_API_VALIDATION)/std_clib/std_clib_main$(EXEEXT) \ + $(ALL_API_VALIDATION)/thread/thread_main$(EXEEXT) \ + $(ALL_API_VALIDATION)/time/time_main$(EXEEXT) \ + $(ALL_API_VALIDATION)/timer/timer_main$(EXEEXT) \ + $(ALL_API_VALIDATION)/traffic_mngr/traffic_mngr_main$(EXEEXT) \ + $(ALL_API_VALIDATION)/shmem/shmem_main$(EXEEXT) \ + $(ALL_API_VALIDATION)/system/system_main$(EXEEXT) \ + ---- With the exception for module pktio, all other modules testing just involves -calling the platform agnostic _main executables (in test/validation). +calling the platform agnostic _main executables (in +/test/common_plat/validation/api). ==== Using other languages The pktio module, above, is actually tested using a bash script. This script is needed to set up the interfaces used by the tests. The `pktio_run` script -eventually calls the platform agnostic `test/validation/pktio/pktio_main` after -setting up the interfaces needed by the tests. -Notice that the path to the script, `pktio/pktio_run`, is pointing to a file -within the tree so is private to this platform. Any -languages supported by the tested platform can be used there, as it will not -impact other platforms. +eventually calls the platform agnostic +`test/common_plat/validation/api/pktio/pktio_main` after setting up the +interfaces needed by the tests. +Notice that the path to the script, `validation/api/pktio/pktio_run.sh`, +is pointing to a file within the tree so is private to +this platform. Any languages supported by the tested platform can be used +there, as it will not impact other platforms. The platform "private" executables (such as this script), of course, must also return one of the return code expected by the automake test harness (0 for success, 77 for skipped, other values for errors). @@ -438,15 +457,20 @@ Sometimes, it may be necessary to call platform specific system calls to check some functionality: For instance, testing `odp_cpumask_*` could involve checking the underlying system CPU mask. On linux, such a test would require using the CPU_ISSET macro, which is linux specific. Such a test would be written in -`/cpumask/...` The contents of this directory would be very -similar to the contents of the platform agnostic side cpu_mask tests -(including a `Makefile.am`), but platform specific test would be written there. +`///cpumask/...` The contents of +this directory would be very similar to the contents of the platform agnostic +side cpu_mask tests (including a `Makefile.am`...), but platform specific test +would be written there. `/Makefile.am` would then trigger the building of the platform specific tests (by listing their module name in `SUBDIRS` and therefore calling the appropriate `Makefile.am`) and then it would call both the platform agnostic executable(s) and the platform specific test executable. -==== Marking tests as inactive +The shm module of the linux-generic ODP API does have a validation +test written this way. You can see it at: +`test/linux-generic/validation/api/shmem` + +==== Marking validation tests as inactive The general policy is that a full run of the validation suite (a `make check`) must pass at all times. However a particular platform may have one or more test @@ -464,7 +488,8 @@ modify the properties of previously registered tests, for example to mark them as inactive. Inactive tests are registered with the test framework but aren't executed and will be recorded as inactive in test reports. -In `test/validation/foo/foo.c`, define all tests for the `foo` module: +In `test/common_plat/validation/api/foo/foo.c`, define all +ivalidation tests for the 'foo' module: [source,c] ------------------ @@ -480,7 +505,7 @@ odp_suiteinfo_t foo_suites[] = { }; ------------------ -In `platform//test/foo/foo_main.c`, register all the tests defined in +In `/validation/api/foo/foo_main.c`, register all the tests defined in the `foo` module, then mark a single specific test case as inactive: [source,c]