From patchwork Tue Aug 4 17:03:21 2015 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Holmes X-Patchwork-Id: 51912 Return-Path: X-Original-To: linaro@patches.linaro.org Delivered-To: linaro@patches.linaro.org Received: from mail-la0-f72.google.com (mail-la0-f72.google.com [209.85.215.72]) by patches.linaro.org (Postfix) with ESMTPS id A59DC22E8E for ; Tue, 4 Aug 2015 17:11:19 +0000 (UTC) Received: by labth1 with SMTP id th1sf6511132lab.2 for ; Tue, 04 Aug 2015 10:11:18 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:delivered-to:delivered-to:from:to:date :message-id:subject:precedence:list-id:list-unsubscribe:list-archive :list-post:list-help:list-subscribe:mime-version:content-type :content-transfer-encoding:errors-to:sender:x-original-sender :x-original-authentication-results:mailing-list; bh=jdnJ6VnYiFhjJbqCZoxAvbPjgP28PuaDzF1nj27Bz/A=; b=X9cFIeJucr7Cavoki60WkDtRF5NER87nUdbZYnDaCF13Wq5qOxHTveosxWwEIS1UCW YlDhDZ/X4HGKpfRboINLCR33kfHYm8HwjgbaffyZbiAjpYEboQ6iomTyRBNLIuTF+ic/ jjyIKwS8XRu/umE3BqyKsG/Mzngp177gWBi6xbFqsL2DMe7BAbKff/APZP/wJT/RVub6 nLCVqb/fqnazF0rYjJw0ZDSc+06cYMW7JSUl12w57yp2mkeAYKYGZk7Prl3YTSN1YWYO ZtzEAb6+Y2Z2xJqEM3pjRKLPPiKyvCm84AuE/IlV/ZmybxVr+gDnhdzm0ldZZOTKCukj CBSQ== X-Gm-Message-State: ALoCoQkbnEvZLYigdeX6eZewHGOktv8tH4fq8rzQi9Vrx3eo+P5m6vQsGBHUYVXwgTyNl7sVHLPo X-Received: by 10.194.80.69 with SMTP id p5mr1431933wjx.2.1438708278606; Tue, 04 Aug 2015 10:11:18 -0700 (PDT) X-BeenThere: patchwork-forward@linaro.org Received: by 10.152.37.196 with SMTP id a4ls69998lak.27.gmail; Tue, 04 Aug 2015 10:11:18 -0700 (PDT) X-Received: by 10.112.13.134 with SMTP id h6mr2990569lbc.15.1438708278281; Tue, 04 Aug 2015 10:11:18 -0700 (PDT) Received: from mail-la0-f41.google.com (mail-la0-f41.google.com. [209.85.215.41]) by mx.google.com with ESMTPS id y9si1123006lbo.119.2015.08.04.10.11.18 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 04 Aug 2015 10:11:18 -0700 (PDT) Received-SPF: pass (google.com: domain of patch+caf_=patchwork-forward=linaro.org@linaro.org designates 209.85.215.41 as permitted sender) client-ip=209.85.215.41; Received: by lady2 with SMTP id y2so7909402lad.0 for ; Tue, 04 Aug 2015 10:11:18 -0700 (PDT) X-Received: by 10.152.1.66 with SMTP id 2mr4747800lak.56.1438708277881; Tue, 04 Aug 2015 10:11:17 -0700 (PDT) X-Forwarded-To: patchwork-forward@linaro.org X-Forwarded-For: patch@linaro.org patchwork-forward@linaro.org Delivered-To: patch@linaro.org Received: by 10.112.7.198 with SMTP id l6csp2536705lba; Tue, 4 Aug 2015 10:11:16 -0700 (PDT) X-Received: by 10.50.80.19 with SMTP id n19mr4671642igx.30.1438708275659; Tue, 04 Aug 2015 10:11:15 -0700 (PDT) Received: from lists.linaro.org (lists.linaro.org. [54.225.227.206]) by mx.google.com with ESMTP id h62si2139642ioh.204.2015.08.04.10.11.14; Tue, 04 Aug 2015 10:11:15 -0700 (PDT) Received-SPF: pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.225.227.206 as permitted sender) client-ip=54.225.227.206; Received: by lists.linaro.org (Postfix, from userid 109) id 451AF61E39; Tue, 4 Aug 2015 17:11:14 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on ip-10-142-244-252.ec2.internal X-Spam-Level: X-Spam-Status: No, score=-2.6 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_LOW, URIBL_BLOCKED autolearn=disabled version=3.4.0 Received: from ip-10-142-244-252.ec2.internal (localhost [127.0.0.1]) by lists.linaro.org (Postfix) with ESMTP id 7634A61E61; Tue, 4 Aug 2015 17:11:01 +0000 (UTC) X-Original-To: lng-odp@lists.linaro.org Delivered-To: lng-odp@lists.linaro.org Received: by lists.linaro.org (Postfix, from userid 109) id 4810861E6F; Tue, 4 Aug 2015 17:10:22 +0000 (UTC) Received: from mail-qg0-f53.google.com (mail-qg0-f53.google.com [209.85.192.53]) by lists.linaro.org (Postfix) with ESMTPS id D231C61E57 for ; Tue, 4 Aug 2015 17:03:27 +0000 (UTC) Received: by qgeh16 with SMTP id h16so11109779qge.3 for ; Tue, 04 Aug 2015 10:03:27 -0700 (PDT) X-Received: by 10.140.35.200 with SMTP id n66mr8183776qgn.21.1438707807410; Tue, 04 Aug 2015 10:03:27 -0700 (PDT) Received: from localhost.localdomain (c-98-221-136-245.hsd1.nj.comcast.net. [98.221.136.245]) by smtp.gmail.com with ESMTPSA id 94sm778444qkz.38.2015.08.04.10.03.26 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Tue, 04 Aug 2015 10:03:26 -0700 (PDT) From: Mike Holmes To: lng-odp@lists.linaro.org Date: Tue, 4 Aug 2015 13:03:21 -0400 Message-Id: <1438707801-6248-1-git-send-email-mike.holmes@linaro.org> X-Mailer: git-send-email 2.1.4 X-Topics: patch Subject: [lng-odp] [PATCH] doc: implementers-guide: add validation description X-BeenThere: lng-odp@lists.linaro.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: , List-Help: , List-Subscribe: , MIME-Version: 1.0 Errors-To: lng-odp-bounces@lists.linaro.org Sender: "lng-odp" X-Removed-Original-Auth: Dkim didn't pass. X-Original-Sender: mike.holmes@linaro.org X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of patch+caf_=patchwork-forward=linaro.org@linaro.org designates 209.85.215.41 as permitted sender) smtp.mail=patch+caf_=patchwork-forward=linaro.org@linaro.org Mailing-list: list patchwork-forward@linaro.org; contact patchwork-forward+owners@linaro.org X-Google-Group-Id: 836684582541 From: Christophe Milard Signed-off-by: Mike Holmes --- Already rendered view http://people.linaro.org/~mike.holmes/implementers-guide/implementers-guide.html Obviously many more sections to add configure.ac | 25 +++ doc/Makefile.am | 6 +- doc/implementers-guide/Makefile.am | 11 ++ doc/implementers-guide/implementers-guide.adoc | 212 +++++++++++++++++++++++++ 4 files changed, 252 insertions(+), 2 deletions(-) create mode 100644 doc/implementers-guide/Makefile.am create mode 100644 doc/implementers-guide/implementers-guide.adoc diff --git a/configure.ac b/configure.ac index 24a3fa1..57d145c 100644 --- a/configure.ac +++ b/configure.ac @@ -224,6 +224,30 @@ fi AM_CONDITIONAL([HAVE_DOXYGEN], [test "x${DOXYGEN}" = "xdoxygen"]) ########################################################################## +# Check for asciidoc availability +########################################################################## +AC_CHECK_PROGS([ASCIIDOC], [asciidoc]) +if test -z "$ASCIIDOC"; + then AC_MSG_WARN([asciidoc not found - continuing without asciidoc support]) +fi + +########################################################################## +# Enable/disable user guide generation +########################################################################## +user_guides=no +AC_ARG_ENABLE([user-guides], + [ --enable-user-guides generate supplemental users guides], + [if test "x$enableval" = "xyes"; then + if test -z "$ASCIIDOC"; + then AC_MSG_ERROR([cannot generate user guides without asciidoc]) + else + user_guides=yes + fi + fi]) + +AM_CONDITIONAL([user_guide], [test "x${user_guides}" = "xyes" ]) + +########################################################################## # Set optional OpenSSL path ########################################################################## AC_ARG_WITH([openssl-path], @@ -288,6 +312,7 @@ AM_CXXFLAGS="-std=c++11" AC_CONFIG_FILES([Makefile doc/Makefile + doc/implementers-guide/Makefile example/Makefile example/classifier/Makefile example/generator/Makefile diff --git a/doc/Makefile.am b/doc/Makefile.am index d2f123a..73bd8e2 100644 --- a/doc/Makefile.am +++ b/doc/Makefile.am @@ -1,6 +1,8 @@ if HAVE_DOXYGEN - - clean-local: rm -rf output endif + +if user_guide +SUBDIRS = implementers-guide +endif diff --git a/doc/implementers-guide/Makefile.am b/doc/implementers-guide/Makefile.am new file mode 100644 index 0000000..9416a6c --- /dev/null +++ b/doc/implementers-guide/Makefile.am @@ -0,0 +1,11 @@ +TARGET = $(top_srcdir)/doc/output/implementers-guide.html + +all-local: $(TARGET) + +$(TARGET): implementers-guide.adoc + @mkdir -p $(top_srcdir)/doc/output + asciidoc --out-file=$@ $< + +clean-local: + rm -f $(TARGET) + diff --git a/doc/implementers-guide/implementers-guide.adoc b/doc/implementers-guide/implementers-guide.adoc new file mode 100644 index 0000000..cd9567c --- /dev/null +++ b/doc/implementers-guide/implementers-guide.adoc @@ -0,0 +1,212 @@ +OpenDataPlane (ODP) Implementers-Guide +======================================= +:toc: + +:numbered!: +[abstract] +Abstract +-------- +This document is intended to guide a new ODP developer. +Further details about ODP may be found at http://opendataplane.org[ODP homepage] + + +:numbered: +The validation Suite +-------------------- +ODP provides a comprehensive set of API validation tests that are intended to be used by implementers during development and by application developers to verify that a particular implementation meets their requirements. + +The list of these tests is expected to grow as ODP grows. + +The list of test executables is run by the automake test harness, when running "make check". +Therefore, as required by this harness, each executable should return 0 on success (tests passed), 77 on inconclusive, or any other values on failure. +The automake functionality shows a status line (PASSED/FAIL...) for each of the ran test executables. + +It is expected that ODP developers will need to run tests as early as possible in the development cycle, before all APIs have been implemented. +Besides, although there are no APIs that are formally listed as optional, it is also expected that there may be cases where a subset of APIs remain unimplemented on a particular platform. +Moreover, some platforms may require specific initialization/termination code prior/after running the standard tests. + +To accommodate with these platform disparities, the ODP validation has been divided in two distinct areas: + +* The platform agnostic area, +* A platform dependent area (one per platform). + +Platform agnostic +~~~~~~~~~~~~~~~~~ +This grouping defines tests that are expected to be executable and succeed on any platform, though possibly with very different performances, depending on the underlying platform. +They are written in plain C code, and may only use functions defined in the standard libC library (besides the ODP functions being tested, of course). +No other languages (like scripting) are allowed as their usage would make assumptions on the platform capability. + +This area is located at: '/test/validation/' + +The ODP API itself is ordered by module, where each module groups the set of ODP API functions related to the same "topic". +Examples of modules includes "classification" (API functions dealing with ingres packets classification), time (functions dealing with time, excluding timers which have their own module), timer,... +The complete module list can be seen at: http://docs.opendataplane.org/linux-generic-doxygen-html/modules.html[ODP Modules] + +Within the platform agnostic area, the tests are also grouped by modules, matching the ODP API modules: '/test/validation/' mainly contains a list of directories matching each module name (as defined by the doxygen "@defgroup" or "@ingroup" statement present in each API ".h" file). + +Within each of these directories, a library (called "lib.la") and its associated ".h" file (called ".h") defines all the test functions for this module as well as few other functions to initialize, terminate, and group the tests. +An executable called "_main*", is also built. It is permissible to generate more than one executable to cover the functionality in the test library for the module. +These executable(s) shall call all the tests for this module. + +See <> for more details. + +It is important to be aware that the tests defined for a given module (defined in 'test/validation/') are focused to test the ODP functions belonging to this module, but are not limited to use this module's ODP functions only: many modules needs some interaction with some other module to be tested. +The obvious illustration of this is for module "init" whose functions are required by all tests of all other modules (as ODP needs to be initialized to test anything else). + + +There is a "Makefile.am" located at the top of the platform agnostic area. Its role is limited to the construction of the different test libraries and the "_main*" executables. No tests are run from this area when "make check" is performed. + +A convenience library '.../test/validation/libodptests.la' (and its associated .h file, '.../test/validation/odptest.h') regrouping all tests symbols of all modules may be built in the future. (The superlib) + +C_UNIT +^^^^^^ +Within a given test executable C_UNIT is used to run the different tests. The usage of C_UNIT implies the following structure: + +* Tests are simple C functions. +* Tests are grouped in arrays called test suites. Each test suite can be associated with a suite initialization/termination function(s), called by C_UNIT before and after the whole suite is ran. +* An array of test suites (and associated init/term functions) defines the test registry ran by the test executable. + +Moreover, two extra functions can be used to initialize/terminate the test executable (these are not part of C_UNIT). + +A test executable return success (0) if every tests of each suite succeed. + +More details about http://cunit.sourceforge.net/doc/index.html[C_Unit users guide] + +[[anchor-1]] +Module test and naming convention +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +* Tests, i.e. C functions which are used in CUNIT test suites are named: + *_test_+++*+++* + + where the suffix idendify the test. + +* Test arrays, i.e. arrays of CU_TestInfo, listing the test functions belonging to a suite, are called: + *_suite+++[_*]+++* + + where the possible suffix can be used if many suites are declared. + +* CUNIT suite init and termination functions are called: + *+++_suite[_*]_init()+++* and *+++_suite[_*]_term()+++* respectively. + + where the possible extra middle pattern can be used if many suites are declared. + +* Suite arrays, i.e. arrays of CU_SuiteInfo used in executables (C_UNIT registry) are called: + *+++_suites[_*]+++* + + where the possible suffix identifies the executable using it, if many. + +* Main executable function(s), are called: + *+++_main[_*]+++* + + where the possible suffix identifies the executable, if many, using it. + +* Init/term functions for the whole executable are called: + *_init* + *_term* + +All the above symbols are part of the generated lib.la libraries. The generated main executable(s) (named _+++main[_*]+++, where the optional suffix is used to distinguish the executables belonging to the same module, if many) simply call(s) the related _main+++[_*]+++ from the library. + +Platform specific +~~~~~~~~~~~~~~~~~ +These tests are located under '/platform//test'. There is one such area for each platform implementing ODP. +This location will be referred as in the rest of this document. + +The normal case +^^^^^^^^^^^^^^^ +If the considered platform needs nothing specific to be tested this directory will contain a single "Makefile.am". +This "Makefile.am" then only lists the executables to be run on "make check" (in the automake TEST variable): when the platform has nothing specific to it, this just list the "_main+++[_*]+++" executables, picked from the platform agnostic area. +For the linux-generic platform, most tested modules fall into this category: currently, the '/platform/linux-generic/test/Makefile.am' looks as follows: + +[source,am] +---- +include $(top_srcdir)/test/Makefile.inc +TESTS_ENVIRONMENT += TEST_DIR=${top_builddir}/test/validation + +ODP_MODULES = pktio + +if test_vald +TESTS = pktio/pktio_run \ + ${top_builddir}/test/validation/buffer/buffer_main$(EXEEXT) \ + ${top_builddir}/test/validation/classification/classification_main$(EXEEXT) \ + ${top_builddir}/test/validation/cpumask/cpumask_main$(EXEEXT) \ + ${top_builddir}/test/validation/crypto/crypto_main$(EXEEXT) \ + ${top_builddir}/test/validation/errno/errno_main$(EXEEXT) \ + ${top_builddir}/test/validation/init/init_main_ok$(EXEEXT) \ + ${top_builddir}/test/validation/init/init_main_abort$(EXEEXT) \ + ${top_builddir}/test/validation/init/init_main_log$(EXEEXT) \ + ${top_builddir}/test/validation/packet/packet_main$(EXEEXT) \ + ${top_builddir}/test/validation/pool/pool_main$(EXEEXT) \ + ${top_builddir}/test/validation/queue/queue_main$(EXEEXT) \ + ${top_builddir}/test/validation/random/random_main$(EXEEXT) \ + ${top_builddir}/test/validation/scheduler/scheduler_main$(EXEEXT) \ + ${top_builddir}/test/validation/synchronizers/synchronizers_main$(EXEEXT) \ + ${top_builddir}/test/validation/thread/thread_main$(EXEEXT) \ + ${top_builddir}/test/validation/time/time_main$(EXEEXT) \ + ${top_builddir}/test/validation/timer/timer_main$(EXEEXT) \ + ${top_builddir}/test/validation/shmem/shmem_main$(EXEEXT) \ + ${top_builddir}/test/validation/system/system_main$(EXEEXT) + +SUBDIRS = $(ODP_MODULES) +endif + +---- + +With the exception for module pktio, all other modules testing just involves calling the platform agnostic _main executables (in test/validation). + +When no platform specific testing is required, the '/Makefile.am' is used to list the tests executables to be run only, as these tests are actually built from the platform agnostic side by the 'test/validation/Makefile.am' (and subdirectories). '/Makefile.am' is involved in building only when platform specific tests exists, as discussed below. + +Using other languages +^^^^^^^^^^^^^^^^^^^^^ +The pktio module, above, is actually tested using a bash script. This script is needed to set up the interfaces used by the tests. The pktio_run script actually eventually calls the platform agnostic 'test/validation/pktio/pktio_main' after setting up the interfaces needed by the tests. +Notice that the path to the script is '/pktio/pktio_run', i.e. it is private to this platform. Any languages supported by the tested platform can be used there, as it will not impact other platforms. +The platform "private" executables (such as this script), of course, must also return one of the return code expected by the automake test harness (0 for success, 77 for inconclusive, other values for errors). + +Defining test wrappers +^^^^^^^^^^^^^^^^^^^^^^ +The pktio case above is actually using a script as wrapper around the "standard" (platform independent) test executable. Wrappers can also be defined by using the LOG_COMPILER variable of automake. +This is applicable in cases where the same wrapper should be used for more then one test, as the test name is passed has parameter to the wrapper. A wrapper is just a program expecting one argument: the test name. + +Automake also supports the usage different wrappers based of the executable filename suffix. See https://www.gnu.org/software/automake/manual/html_node/Parallel-Test-Harness.html[Parallel-Test-Harness] for more information. + +To add a wrapper around the executed test, just add the following LOG_COMPILER definition line in the '/Makefile.am': + +[source,am] +---- +... +if test_vald +LOG_COMPILER = $(top_srcdir)/platform/linux-generic/test/wrapper-script +TESTS = pktio/pktio_run \ +... +---- + +Here follows a dummy example of what wrapper-script could be: + +[source,bash] +---- +#!/bin/bash + +# The parameter, $1, is the name of the test executable to run +echo "WRAPPER!!!" +echo "running $1!" + +# run the test: +$1 +# remember the test result: +res=$? + +echo "Do something to clean up the mess here :-)" +# return the test result. +exit $res +---- + +Note how the above script stores the return code of the test executable to return it properly to the automake test harness. + +Defining platform specific tests +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +Sometimes, it may be necessary to call platform specific system calls to check some functionality: For instance, testing odp_cpumask_* could involve checking the underlying system CPU mask. On linux, such a test would require using the CPU_ISSET macro, which is linux specific. Such a test would be written in '/cpumask/...' The contents of this directory would be very similar to the contents of the platform agnostic side cpu_mask tests (including a Makefile.am...), but platform specific test would be written there. +'/Makefile.am' would then trigger the building of the platform specific tests (by listing their module name in SUBDIRS and therefore calling the appropriate Makefile.am) and then it would call both the platform agnostic executable(s) and the platform specific test executable. + +Skipping tests during development +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +During ODP development, it may be useful to skip some test. This can be achieved by creating a new test executable (still on the platform side), picking up the required tests from the platform agnostic lib.la. + +The top Makefile would then call only the platform specific executable, hence skipping the tests which have been omitted. + +TIP: You probably want to copy the platform-agnostic module main function and prune it from the undesired tests when writing your own platform specific main, for a given module. + +Permanently skipping test +^^^^^^^^^^^^^^^^^^^^^^^^^^ +If a platform wants to permanently skip a test (i.e. a part of the ODP API is and will not be supported on that platform), it is recommended to use the function odp_cunit_TBD() to removed the tests or suite from the list of tests. This gives a chance to the test environment to trace this removal. +