From patchwork Fri Sep 14 15:16:22 2012 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andy Doan X-Patchwork-Id: 11422 Return-Path: X-Original-To: patchwork@peony.canonical.com Delivered-To: patchwork@peony.canonical.com Received: from fiordland.canonical.com (fiordland.canonical.com [91.189.94.145]) by peony.canonical.com (Postfix) with ESMTP id CB05723E2F for ; Fri, 14 Sep 2012 15:16:51 +0000 (UTC) Received: from mail-ie0-f180.google.com (mail-ie0-f180.google.com [209.85.223.180]) by fiordland.canonical.com (Postfix) with ESMTP id 1DC3AA3556E for ; Fri, 14 Sep 2012 15:16:26 +0000 (UTC) Received: by ieak11 with SMTP id k11so6754752iea.11 for ; Fri, 14 Sep 2012 08:16:25 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=x-forwarded-to:x-forwarded-for:delivered-to:received-spf :content-type:mime-version:x-launchpad-project:x-launchpad-branch :x-launchpad-message-rationale:x-launchpad-branch-revision-number :x-launchpad-notification-type:to:from:subject:message-id:date :reply-to:sender:errors-to:precedence:x-generated-by :x-launchpad-hash:x-gm-message-state; bh=+jYC1xM/GSNXjEZMuE2MetTIx0v+1CifYZKZ7Lcta/8=; b=EYg2JE/x4hBju4DsRplOiyckr7uHS9MdwK4ycF/YqBwaRsuoT3rZxJxY/rFFeHZHkg TSQIulc3HtoHKxRfSzJ1wr2sY7LsHzkzgCLX72Cj7d+dm7WzZ1k5e9BuhE2eKhZR3vHs Ki7Cc6cjzWAQuPlG8qu65Tj2wV0jbK8zGG8J8juJvZiUJNJ/2NZOLLKTY1DlGVQa5FP5 yJX/etULEn1k+r6cs1K4Z1EmeOaZmClo1nVgy+v3urwNVlpsGOnki3LRPU4AylFSus7Q gc8/G4FJ63xfgs9Ta70qMsBzbeiwvm5xAGYIRtsRAOjCpg2QHd1Z1xmbiEJR8hVEYGv1 gerA== Received: by 10.50.242.3 with SMTP id wm3mr29150875igc.0.1347635785443; Fri, 14 Sep 2012 08:16:25 -0700 (PDT) X-Forwarded-To: linaro-patchwork@canonical.com X-Forwarded-For: patch@linaro.org linaro-patchwork@canonical.com Delivered-To: patches@linaro.org Received: by 10.50.184.232 with SMTP id ex8csp195404igc; Fri, 14 Sep 2012 08:16:24 -0700 (PDT) Received: by 10.180.107.163 with SMTP id hd3mr6539077wib.19.1347635783276; Fri, 14 Sep 2012 08:16:23 -0700 (PDT) Received: from indium.canonical.com (indium.canonical.com. [91.189.90.7]) by mx.google.com with ESMTPS id b4si4489313wiz.1.2012.09.14.08.16.22 (version=TLSv1/SSLv3 cipher=OTHER); Fri, 14 Sep 2012 08:16:23 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of bounces@canonical.com designates 91.189.90.7 as permitted sender) client-ip=91.189.90.7; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of bounces@canonical.com designates 91.189.90.7 as permitted sender) smtp.mail=bounces@canonical.com Received: from ackee.canonical.com ([91.189.89.26]) by indium.canonical.com with esmtp (Exim 4.71 #1 (Debian)) id 1TCXd0-0005FU-Bg for ; Fri, 14 Sep 2012 15:16:22 +0000 Received: from ackee.canonical.com (localhost [127.0.0.1]) by ackee.canonical.com (Postfix) with ESMTP id 4DFACE0148 for ; Fri, 14 Sep 2012 15:16:22 +0000 (UTC) MIME-Version: 1.0 X-Launchpad-Project: lava-test X-Launchpad-Branch: ~linaro-validation/lava-test/trunk X-Launchpad-Message-Rationale: Subscriber X-Launchpad-Branch-Revision-Number: 173 X-Launchpad-Notification-Type: branch-revision To: Linaro Patch Tracker From: noreply@launchpad.net Subject: [Branch ~linaro-validation/lava-test/trunk] Rev 173: doc updates for usage Message-Id: <20120914151622.24059.91758.launchpad@ackee.canonical.com> Date: Fri, 14 Sep 2012 15:16:22 -0000 Reply-To: noreply@launchpad.net Sender: bounces@canonical.com Errors-To: bounces@canonical.com Precedence: bulk X-Generated-By: Launchpad (canonical.com); Revision="15944"; Instance="launchpad-lazr.conf" X-Launchpad-Hash: 647a93ebae51354079fdbf5ad93d14a844fbf978 X-Gm-Message-State: ALoCoQmF47J57U4YE8My82E8Uasyi9i1IPwdF+YV34RCX2/nQmiBimrx+WB1E8RNF3OCYe9g1cFm Merge authors: Andy Doan (doanac) Related merge proposals: https://code.launchpad.net/~doanac/lava-test/doc-updates/+merge/124045 proposed by: Andy Doan (doanac) ------------------------------------------------------------ revno: 173 [merge] committer: Andy Doan branch nick: lt timestamp: Fri 2012-09-14 10:15:00 -0500 message: doc updates for usage modified: doc/usage.rst --- lp:lava-test https://code.launchpad.net/~linaro-validation/lava-test/trunk You are subscribed to branch lp:lava-test. To unsubscribe from this branch go to https://code.launchpad.net/~linaro-validation/lava-test/trunk/+edit-subscription === modified file 'doc/usage.rst' --- doc/usage.rst 2012-03-31 08:44:50 +0000 +++ doc/usage.rst 2012-09-12 20:08:31 +0000 @@ -60,76 +60,26 @@ By default parse will print the bundle to standard output for inspection. It should be redirected to a pager for easier verification. -.. note:: - - While the syntax of the bundle created with `lava-test parse` is always - correct (or, if the parser does something really, really strange, a - detailed error is reported) the actual contents may not be what you - intended it to be. Parsers are ultimately fragile as they mostly deal with - unstructured or semi-structured free-form text that most test programs seem - to produce. The ultimate goal of a developer should be to produce - unambiguous, machine readable format. This level of integration would allow - to wrap a whole class of tests in one go (such as all xUnit-XML speaking - test frameworks). - -Usage with the dispatcher -^^^^^^^^^^^^^^^^^^^^^^^^^ - -The dispatcher is useful for automating LAVA Test environment setup, describing -test scenarios (the list of tests to invoke) and finally storing the results in -the LAVA dashboard. - -Typically this mode is based on the following sequence of commands: - -#. Install lava-test (from PPA or source) along with the required dependencies -#. (optional) for out of tree tests install the additional `test definition` package -#. Install the test or tests that are to be invoked with ``lava-tool install``. -#. Run, parse and store in one go with ``lava-tool run --output=FILE``. - -Here the whole setup is non-interactive and at the end the dispatcher can copy -the output bundle for additional processing. - -Automation considerations -^^^^^^^^^^^^^^^^^^^^^^^^^ - .. _wrapping_existing_test_or_benchmark: Wrapping existing test or benchmark =================================== -LAVA Test can be extended in several different ways. There is no best method, -each has some pros and cons. In general we welcome any freely redistributable, -generic tests. Those enrich the LAVA ecosystem and by providing useful -out-of-the-box features to our users. - -Technically all tests are hidden behind a set of abstract interfaces that tell -LAVA Test what to do in response to operator or dispatcher actions. The primary -interface is :class:`~lava_test.api.core.ITest` and the three principal -methods: :meth:`~lava_test.api.core.ITest.install`, -:meth:`~lava_test.api.core.ITest.run`, -:meth:`~lava_test.api.core.ITest.parse`. - -In practice it is usually much easier to instantiate our pluggable delegate -test (:class:`lava_test.core.tests.Test`) and define the three delegates that -know how to install, run and parse. Again for each step we have a base class -that can be easily customized or even used directly as is. Those classes are -:class:`~lava_test.core.installers.TestInstaller`, -:class:`~lava_test.core.runners.TestRunner` and -:class:`~lava_test.core.parsers.TestParser`. They all implement well-defined -interfaces (specified in :mod:`lava_test.api.delegates`) so if you wish to -customize them you should become familiar with the API requirements first. - -Contributing new tests to LAVA -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +There are three different ways you can choose from to add a test: + + * in-tree test + * out-of-tree + * declarative test + +In-Tree Test +^^^^^^^^^^^^ The most direct way to add a new test is to contribute patches to LAVA Test itself. This method will simply add a new test definition to the collection of available tests. This method is recommended for generic tests that rarely change and are -suitable for wide variety of hardware and software (assuming basic Linux-like -system, Android tests are a special case). - +suitable for wide variety of hardware and software. The advantage is that those tests can be invoked out of the box and will be maintained by the LAVA team. The disadvantage is that all changes to those tests need to follow Linaro development work flow, get reviewed and finally @@ -138,7 +88,7 @@ Test definitions are simply a way of telling LAVA-Test how to install a test, run it, and interpret the results. Tests definitions are in a simplified python format, and can be as simple as a few lines. More advanced test definitions can -be written by deriving from the base classes. +be written by deriving from the base classes. Defining a simple test ++++++++++++++++++++++ @@ -283,8 +233,8 @@ common features are found that would make it possible to eliminate or simplify cases like this, they should be merged into the Lava-test libraries. -Maintaining out-of-tree tests -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +Out-Of-Tree Test +^^^^^^^^^^^^^^^^ For some kinds of tests (proprietary, non-generic, in rapid development, fused with application code) contributing their definition to upstream LAVA Test @@ -317,21 +267,76 @@ will discover this entry point, import the relevant module and make the test definition available. -Maintaining simple declarative tests -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -By registering pure declarative tests at runtime. - -.. todo:: - - Describe how to use declarative tests. It would be a nice - extension of the tutorial once the user feels comfortable with - the initial python-based version. - -Writing new tests from scratch +Declarative Test +^^^^^^^^^^^^^^^^ + +As you can in the most simple example of an in-tree test, there is very little +python required of the user. Declarative tests allow you to define these bits +in a JSON format that LAVA can use. This can be a nice alternative to an +out-of-tree test. Some examples of declarative tests are: + +Pass/Fail ++++++++++ +If you have a test that prints out PASS/FAIL messages, your json could be:: + + { + "format": "Lava-Test Test Definition 1.0", + "test_id": "bigLITTLE", + "run": { + "steps": ["/bin/echo cache-coherency-switching : FAIL"] + }, + "parse": { + "pattern": "(?P.*-*)\\s+:\\s+(?P(PASS|FAIL))", + "fixupdict": { + "PASS": "pass", + "FAIL": "fail" + } + } + } + +Stream +++++++ +This does the equivalent of the in-tree stream test:: + + { + "format": "LAVA-Test Test Definition Format", + "test_id": "stream-json", + "install": { + "url": "http://www.cs.virginia.edu/stream/FTP/Code/stream.c", + "steps": ["cc stream.c -O2 -fopenmp -o stream"] + }, + "run": { + "steps": ["./stream"] + }, + "parse": { + "pattern": "^(?P\\w+):\\W+(?P\\d+\\.\\d+)", + "appendall": { + "units": "MB/s", + "result": "pass" + } + } + } + +Executing a Declarative Test +++++++++++++++++++++++++++++ + +The main difference with declarative tests is that you must register them with +lava-test so that it knows about it. You can do this with:: + + lava-test register-test $TEST_DEF_URL + +Then you do the standard:: + + lava-test install + lava-test run + +Writing new tests from scratch ============================== -.. todo:: +The thing to keep in mind with lava-test is that its not intended to be a +test framework itself. So the best advice is to build your test the best way +you see fit. Then make a thin wrapper for lava-test using one the three methods +described in the previous section. - Describe considerations for test writers. Using native test - format with human-readable output adapters. +The most important thing is to make your tests output its results on a single +line. This makes declaring the parsing for lava-test much easier.