=== added file 'doc/external_measurement.rst'
@@ -0,0 +1,110 @@
+Hooks, Signals and External Measurement
+=======================================
+
+.. warning::
+ This is work in progress! Expect changes in details until at least early 2013.
+
+It is sometimes the case that an interesting test cannot be run solely
+on the device being tested: additional data from somewhere else is
+required. For example, a test of the sound subsystem may want to
+generate audio, play it, capture it on another system and then compare
+the generated and captured audio. A `lava-test-shell`_ test can be
+written to send **signals** to indicate when a test case starts and
+finishes which can be handled by a **handler** specified by the test
+definition.
+
+.. _`lava-test-shell`: lava_test_shell.html
+
+Signals
+-------
+
+A signal is a message from the system being tested ("device") to the
+system the dispatcher is running on ("host"). The messaging is
+synchronous and uni-directional: lava-test-shell on the device will
+wait for the signal to be processesed and there is no way for the
+device to receieve data from the host.
+
+Generally speaking, we expect a test author will only be interested in
+handling the "start test case" and "end test case" signals that are
+sent by ``lava-test-case --shell``.
+
+Handler
+-------
+
+A handler is a Python class that subclasses:
+
+.. autoclass:: lava_dispatcher.signals.SignalHandler
+
+This class defines three methods that you almost certainly want to
+override:
+
+ 1. ``start_testcase(self, test_case_id):``
+
+ Called when a testcase starts on the device. The return value of
+ this method is passed to both ``end_testcase`` and
+ ``processes_test_run``.
+
+ The expected case is something like: starting a process that
+ captures some data from or about the device and returning a
+ dictionary that indicates the pid of that process and where its
+ output is going.
+
+ 2. ``end_testcase(self, test_case_id, case_data):``
+
+ Called when a testcase ends on the device. ``case_data`` is
+ whatever the corresponding ``start_testcase`` call returned.
+
+ The expected case here is that you will terminate the process that
+ was started by ``start_testcase``.
+
+ 3. ``process_test_result(self, test_result, case_data):``
+
+ Here you are expected to add the data that was recorded during the
+ test run to the results. You need to know about the bundle format
+ to do this.
+
+These methods are invoked with catch-all exception handlers around
+them so you don't have to be super careful in their implementation: it
+should not be possible to crash the whole dispatcher with a typo in
+one of them.
+
+Here is a very simple complete handler::
+
+ import datetime
+ import time
+
+ from json_schema_validator.extensions import timedelta_extension
+
+ from lava_dispatcher.signals import SignalHandler
+
+ class AddDuration(SignalHandler):
+
+ def start_testcase(self, test_case_id):
+ return {
+ 'starttime': time.time()
+ }
+
+ def end_testcase(self, test_case_id, data):
+ data['endtime'] = time.time()
+
+ def postprocess_test_result(self, test_result, data):
+ delta = datetime.timedelta(seconds=data['endtime'] - data['starttime'])
+ test_result['duration'] = timedelta_extension.to_json(delta)
+
+Specifying a handler
+--------------------
+
+A handlers are named the test definition, for example::
+
+ handler:
+ handler-name: add-duration
+
+The name is the name of an `entry point`_ from the
+``lava.signal_handlers`` "group". The entry point must be provided by
+a package installed into the instance that the dispatcher is running
+from.
+
+.. _`entry point`: http://packages.python.org/distribute/pkg_resources.html#entry-points
+
+We will soon provide a way to bundle the signal handler along with the
+test definition.
=== modified file 'doc/index.rst'
@@ -33,6 +33,8 @@
configuration.rst
jobfile.rst
usage.rst
+ lava_test_shell.rst
+ external_measurement.rst
proxy.rst
* :ref:`search`
=== modified file 'doc/lava_test_shell.rst'
@@ -55,6 +55,11 @@
* ``lava-test-case``
* ``lava-test-case-attach``
+You need to use ``lava-test-case`` (specifically, ``lava-test-case
+--shell``) when you are working with `hooks, signals and external
+measurement`_.
+
+.. _`hooks, signals and external measurement`: external_measurement.html
lava-test-case
--------------
@@ -89,6 +94,9 @@
- "lava-test-case fail-test --shell false"
- "lava-test-case pass-test --shell true"
+The --shell form also sends the start test case and end test case
+signals that are described in `hooks, signals and external
+measurement`_.
lava-test-case-attach
---------------------
@@ -107,6 +115,7 @@
3. (optional) the MIME type of the file (if no MIME type is passed, a
guess is made based on the filename)
+
Handling Dependencies (Ubuntu)
==============================
=== modified file 'lava_dispatcher/actions/lava_test_shell.py'
@@ -45,9 +45,13 @@
# for test authors.
# lava-test-runner The job that runs the tests on boot.
# lava-test-shell A helper to run a test suite.
+# lava-test-case A helper to record information about a test
+# result.
# lava-test-case-attach A helper to attach a file to a test result.
# tests/
# ${IDX}_${TEST_ID}/ One directory per test to be executed.
+# uuid The "analyzer_assigned_uuid" of the
+# test_run that is being generated.
# testdef.yml The test definition.
# install.sh The install steps.
# run.sh The run steps.
@@ -71,14 +75,15 @@
# ${IDX}_${TEST_ID}-${TIMESTAMP}/
# testdef.yml
# stdout.log
-# return_code The exit code of run.sh.
+# return_code The exit code of run.sh.
+# analyzer_assigned_uuid
# attachments/
# install.sh
# run.sh
# ${FILENAME} The attached data.
# ${FILENAME}.mimetype The mime type of the attachment.
-# attributes/
-# ${ATTRNAME} Content is value of attribute
+# attributes/
+# ${ATTRNAME} Content is value of attribute
# tags/
# ${TAGNAME} Content of file is ignored.
# results/
@@ -98,26 +103,35 @@
# After the test run has completed, the /lava/results directory is pulled over
# to the host and turned into a bundle for submission to the dashboard.
-import yaml
from glob import glob
-import time
import logging
import os
import pexpect
+import pkg_resources
import shutil
import stat
import subprocess
import tempfile
+import time
+from uuid import uuid4
+
+import yaml
from linaro_dashboard_bundle.io import DocumentIO
import lava_dispatcher.lava_test_shell as lava_test_shell
-import lava_dispatcher.utils as utils
+from lava_dispatcher.signals import SignalDirector
+from lava_dispatcher import utils
from lava_dispatcher.actions import BaseAction
from lava_dispatcher.device.target import Target
from lava_dispatcher.downloader import download_image
+# Reading from STDIN in the lava-test-shell doesn't work well because its
+# STDIN is /dev/console which we are doing echo's on in our scripts. This
+# just makes a well known fifo we can read the ACK's with
+ACK_FIFO = '/lava_ack.fifo'
+
LAVA_TEST_DIR = '%s/../../lava_test_shell' % os.path.dirname(__file__)
LAVA_TEST_ANDROID = '%s/lava-test-runner-android' % LAVA_TEST_DIR
LAVA_TEST_UBUNTU = '%s/lava-test-runner-ubuntu' % LAVA_TEST_DIR
@@ -139,7 +153,7 @@
Target.ubuntu_deployment_data['lava_test_shell'] = LAVA_TEST_SHELL
Target.ubuntu_deployment_data['lava_test_case'] = LAVA_TEST_CASE
Target.ubuntu_deployment_data['lava_test_case_attach'] = LAVA_TEST_CASE_ATTACH
-Target.ubuntu_deployment_data['lava_test_sh_cmd'] = '/bin/sh'
+Target.ubuntu_deployment_data['lava_test_sh_cmd'] = '/bin/bash'
Target.ubuntu_deployment_data['lava_test_dir'] = '/lava'
Target.ubuntu_deployment_data['lava_test_results_part_attr'] = 'root_part'
@@ -183,6 +197,255 @@
Target.android_deployment_data['lava_test_configure_startup'] = \
_configure_android_startup
+def _get_testdef_git_repo(testdef_repo, tmpdir, revision):
+ cwd = os.getcwd()
+ gitdir = os.path.join(tmpdir, 'gittestrepo')
+ try:
+ subprocess.check_call(['git', 'clone', testdef_repo,
+ gitdir])
+ if revision:
+ os.chdir(gitdir)
+ subprocess.check_call(['git', 'checkout', revision])
+ return gitdir
+ except Exception as e:
+ logging.error('Unable to get test definition from git\n' + str(e))
+ finally:
+ os.chdir(cwd)
+
+
+def _get_testdef_bzr_repo(testdef_repo, tmpdir, revision):
+ bzrdir = os.path.join(tmpdir, 'bzrtestrepo')
+ try:
+ # As per bzr revisionspec, '-1' is "The last revision in a
+ # branch".
+ if revision is None:
+ revision = '-1'
+
+ subprocess.check_call(
+ ['bzr', 'branch', '-r', revision, testdef_repo, bzrdir],
+ env={'BZR_HOME': '/dev/null', 'BZR_LOG': '/dev/null'})
+ return bzrdir
+ except Exception as e:
+ logging.error('Unable to get test definition from bzr\n' + str(e))
+
+
+class TestDefinitionLoader(object):
+ """
+ A TestDefinitionLoader knows how to load test definitions from the data
+ provided in the job file.
+ """
+
+ def __init__(self, context, tmpbase):
+ self.testdefs = []
+ self.context = context
+ self.tmpbase = tmpbase
+ self.testdefs_by_uuid = {}
+
+ def _append_testdef(self, testdef_obj):
+ testdef_obj.load_signal_handler()
+ self.testdefs.append(testdef_obj)
+ self.testdefs_by_uuid[testdef_obj.uuid] = testdef_obj
+
+ def load_from_url(self, url):
+ tmpdir = utils.mkdtemp(self.tmpbase)
+ testdef_file = download_image(url, self.context, tmpdir)
+ with open(testdef_file, 'r') as f:
+ logging.info('loading test definition')
+ testdef = yaml.load(f)
+
+ idx = len(self.testdefs)
+
+ self._append_testdef(URLTestDefinition(idx, testdef))
+
+ def load_from_repo(self, testdef_repo):
+ tmpdir = utils.mkdtemp(self.tmpbase)
+ if 'git-repo' in testdef_repo:
+ repo = _get_testdef_git_repo(
+ testdef_repo['git-repo'], tmpdir, testdef_repo.get('revision'))
+ name = os.path.splitext(os.path.basename(testdef_repo['git-repo']))[0]
+ info = _git_info(testdef_repo['git-repo'], repo, name)
+
+ if 'bzr-repo' in testdef_repo:
+ repo = _get_testdef_bzr_repo(
+ testdef_repo['bzr-repo'], tmpdir, testdef_repo.get('revision'))
+ name = testdef_repo['bzr-repo'].replace('lp:', '').split('/')[-1]
+ info = _bzr_info(testdef_repo['bzr-repo'], repo, name)
+
+ test = testdef_repo.get('testdef', 'lavatest.yaml')
+ with open(os.path.join(repo, test), 'r') as f:
+ logging.info('loading test definition ...')
+ testdef = yaml.load(f)
+
+ idx = len(self.testdefs)
+ self._append_testdef(RepoTestDefinition(idx, testdef, repo, info))
+
+
+def _bzr_info(url, bzrdir, name):
+ cwd = os.getcwd()
+ try:
+ os.chdir('%s' % bzrdir)
+ revno = subprocess.check_output(['bzr', 'revno']).strip()
+ return {
+ 'project_name': name,
+ 'branch_vcs': 'bzr',
+ 'branch_revision': revno,
+ 'branch_url': url,
+ }
+ finally:
+ os.chdir(cwd)
+
+
+def _git_info(url, gitdir, name):
+ cwd = os.getcwd()
+ try:
+ os.chdir('%s' % gitdir)
+ commit_id = subprocess.check_output(
+ ['git', 'log', '-1', '--pretty=%H']).strip()
+ return {
+ 'project_name': name,
+ 'branch_vcs': 'git',
+ 'branch_revision': commit_id,
+ 'branch_url': url,
+ }
+ finally:
+ os.chdir(cwd)
+
+
+class URLTestDefinition(object):
+ """
+ A test definition that was loaded from a URL.
+ """
+
+ def __init__(self, idx, testdef):
+ self.testdef = testdef
+ self.idx = idx
+ self.test_run_id = '%s_%s' % (idx, self.testdef['metadata']['name'])
+ self.uuid = str(uuid4())
+ self._sw_sources = []
+ self.handler = None
+
+ def load_signal_handler(self):
+ hook_data = self.testdef.get('handler')
+ if not hook_data:
+ return
+ try:
+ handler_name = hook_data['handler-name']
+ logging.info("Loading handler named %s", handler_name)
+ handler_eps = list(
+ pkg_resources.iter_entry_points(
+ 'lava.signal_handlers', handler_name))
+ if len(handler_eps) == 0:
+ logging.error("No handler named %s found", handler_name)
+ return
+ elif len(handler_eps) > 1:
+ logging.warning(
+ "Multiple handlers named %s found. Picking one arbitrarily.",
+ handler_name)
+ handler_ep = handler_eps[0]
+ logging.info("Loading handler from %s" % handler_ep.dist)
+ handler_cls = handler_ep.load()
+ self.handler = handler_cls(self, **hook_data.get('params', {}))
+ except Exception:
+ logging.exception("loading handler failed")
+
+ def _create_repos(self, testdir):
+ cwd = os.getcwd()
+ try:
+ os.chdir(testdir)
+
+ for repo in self.testdef['install'].get('bzr-repos', []):
+ logging.info("bzr branch %s" % repo)
+ # Pass non-existent BZR_HOME value, or otherwise bzr may
+ # have non-reproducible behavior because it may rely on
+ # bzr whoami value, presence of ssh keys, etc.
+ subprocess.check_call(['bzr', 'branch', repo],
+ env={'BZR_HOME': '/dev/null', 'BZR_LOG': '/dev/null'})
+ name = repo.replace('lp:', '').split('/')[-1]
+ self._sw_sources.append(_bzr_info(repo, name, name))
+
+ for repo in self.testdef['install'].get('git-repos', []):
+ logging.info("git clone %s" % repo)
+ subprocess.check_call(['git', 'clone', repo])
+ name = os.path.splitext(os.path.basename(repo))[0]
+ self._sw_sources.append(_git_info(repo, name, name))
+ finally:
+ os.chdir(cwd)
+
+ def _create_target_install(self, hostdir, targetdir):
+ with open('%s/install.sh' % hostdir, 'w') as f:
+ f.write('set -ex\n')
+ f.write('cd %s\n' % targetdir)
+
+ # TODO how should we handle this for Android?
+ deps = self.testdef['install'].get('deps', [])
+ if deps:
+ f.write('sudo apt-get update\n')
+ f.write('sudo apt-get install -y ')
+ for dep in deps:
+ f.write('%s ' % dep)
+ f.write('\n')
+
+ steps = self.testdef['install'].get('steps', [])
+ if steps:
+ for cmd in steps:
+ f.write('%s\n' % cmd)
+
+ def copy_test(self, hostdir, targetdir):
+ """Copy the files needed to run this test to the device.
+
+ :param hostdir: The location on the device filesystem to copy too.
+ :param targetdir: The location `hostdir` will have when the device
+ boots.
+ """
+ utils.ensure_directory(hostdir)
+ with open('%s/testdef.yaml' % hostdir, 'w') as f:
+ f.write(yaml.dump(self.testdef))
+
+ with open('%s/uuid' % hostdir, 'w') as f:
+ f.write(self.uuid)
+
+ if 'install' in self.testdef:
+ self._create_repos(hostdir)
+ self._create_target_install(hostdir, targetdir)
+
+ with open('%s/run.sh' % hostdir, 'w') as f:
+ f.write('set -e\n')
+ f.write('export TESTRUN_ID=%s\n' % self.test_run_id)
+ f.write('[ -p %s ] && rm %s\n' % (ACK_FIFO, ACK_FIFO))
+ f.write('mkfifo %s\n' % ACK_FIFO)
+ f.write('cd %s\n' % targetdir)
+ f.write('UUID=`cat uuid`\n')
+ f.write('echo "<LAVA_SIGNAL_STARTRUN $TESTRUN_ID $UUID>"\n')
+ f.write('#wait up to 10 minutes for an ack from the dispatcher\n')
+ f.write('read -t 600 < %s\n' % ACK_FIFO)
+ steps = self.testdef['run'].get('steps', [])
+ if steps:
+ for cmd in steps:
+ f.write('%s\n' % cmd)
+ f.write('echo "<LAVA_SIGNAL_ENDRUN $TESTRUN_ID $UUID>"\n')
+ f.write('#wait up to 10 minutes for an ack from the dispatcher\n')
+ f.write('read -t 600 < %s\n' % ACK_FIFO)
+
+
+class RepoTestDefinition(URLTestDefinition):
+ """
+ A test definition that was loaded from a VCS repository.
+
+ The difference is that the files from the repository are also copied to
+ the device.
+ """
+
+ def __init__(self, idx, testdef, repo, info):
+ URLTestDefinition.__init__(self, idx, testdef)
+ self.repo = repo
+ self._sw_sources.append(info)
+
+ def copy_test(self, hostdir, targetdir):
+ URLTestDefinition.copy_test(self, hostdir, targetdir)
+ for filepath in glob(os.path.join(self.repo, '*')):
+ shutil.copy2(filepath, hostdir)
+ logging.info('copied all test files')
+
class cmd_lava_test_shell(BaseAction):
@@ -216,93 +479,44 @@
target = self.client.target_device
self._assert_target(target)
- self._configure_target(target, testdef_urls, testdef_repos)
+ testdefs_by_uuid = self._configure_target(target, testdef_urls, testdef_repos)
+
+ signal_director = SignalDirector(self.client, testdefs_by_uuid)
with target.runner() as runner:
- patterns = [
+ start = time.time()
+ while self._keep_running(runner, timeout, signal_director):
+ elapsed = time.time() - start
+ timeout = int(timeout - elapsed)
+
+ self._bundle_results(target, signal_director, testdefs_by_uuid)
+
+ def _keep_running(self, runner, timeout, signal_director):
+ patterns = [
'<LAVA_TEST_RUNNER>: exiting',
pexpect.EOF,
pexpect.TIMEOUT,
+ '<LAVA_SIGNAL_(\S+) ([^>]+)>',
]
- idx = runner._connection.expect(patterns, timeout=timeout)
- if idx == 0:
- logging.info('lava_test_shell seems to have completed')
- elif idx == 1:
- logging.warn('lava_test_shell connection dropped')
- elif idx == 2:
- logging.warn('lava_test_shell has timed out')
-
- self._bundle_results(target)
-
- def _get_test_definition(self, d, ldir, testdef_src, tmpdir, isrepo=False):
- repo = None
- test = 'lavatest.yaml'
-
- if isrepo:
- if 'git-repo' in testdef_src:
- repo = self._get_testdef_git_repo(testdef_src['git-repo'],
- tmpdir,
- testdef_src.get('revision'))
-
- if 'bzr-repo' in testdef_src:
- repo = self._get_testdef_bzr_repo(testdef_src['bzr-repo'],
- tmpdir,
- testdef_src.get('revision'))
-
- if 'testdef' in testdef_src:
- test = testdef_src['testdef']
-
- with open(os.path.join(repo, test), 'r') as f:
- testdef = yaml.load(f)
- else:
- test = download_image(testdef_src, self.context, tmpdir)
- with open(test, 'r') as f:
- testdef = yaml.load(f)
-
- logging.info('loaded test definition ...')
-
- # android mount the partition under /system, while ubuntu mounts under
- # /, so we have hdir for where it is on the host and tdir for how the
- # target will see the path
- timestamp = str(time.time())
- hdir = '%s/tests/%s_%s' % (d, timestamp, testdef['metadata']['name'])
- tdir = '%s/tests/%s_%s' % (ldir, timestamp, testdef['metadata']['name'])
- self._copy_test(hdir, tdir, testdef, repo)
-
- return tdir
-
- def _get_testdef_git_repo(self, testdef_repo, tmpdir, revision):
- cwd = os.getcwd()
- gitdir = os.path.join(tmpdir, 'gittestrepo')
- try:
- subprocess.check_call(['git', 'clone', testdef_repo, gitdir])
- if revision:
- os.chdir(gitdir)
- subprocess.check_call(['git', 'checkout', revision])
- return gitdir
- except Exception as e:
- logging.error('Unable to get test definition from git\n' + str(e))
- finally:
- os.chdir(cwd)
-
- def _get_testdef_bzr_repo(self, testdef_repo, tmpdir, revision):
- bzrdir = os.path.join(tmpdir, 'bzrtestrepo')
- try:
- # As per bzr revisionspec, '-1' is "The last revision in a
- # branch".
- if revision is None:
- revision = '-1'
-
- # Pass non-existent BZR_HOME value, or otherwise bzr may
- # have non-reproducible behavior because it may rely on
- # bzr whoami value, presence of ssh keys, etc.
- subprocess.check_call(
- ['bzr', 'branch', '-r', revision, testdef_repo, bzrdir],
- env={'BZR_HOME': '/dev/null', 'BZR_LOG': '/dev/null'})
-
- return bzrdir
- except Exception as e:
- logging.error('Unable to get test definition from bzr\n' + str(e))
+
+ idx = runner._connection.expect(patterns, timeout=timeout)
+ if idx == 0:
+ logging.info('lava_test_shell seems to have completed')
+ elif idx == 1:
+ logging.warn('lava_test_shell connection dropped')
+ elif idx == 2:
+ logging.warn('lava_test_shell has timed out')
+ elif idx == 3:
+ name, params = runner._connection.match.groups()
+ params = params.split()
+ try:
+ signal_director.signal(name, params)
+ except:
+ logging.exception("on_signal failed")
+ runner._connection.sendline('echo LAVA_ACK > %s' % ACK_FIFO)
+ return True
+
+ return False
def _copy_runner(self, mntdir, target):
runner = target.deployment_data['lava_test_runner']
@@ -311,7 +525,7 @@
shcmd = target.deployment_data['lava_test_sh_cmd']
- for key in ['lava_test_shell', 'lava_test_case', 'lava_test_case_attach']:
+ for key in ['lava_test_shell', 'lava_test_case_attach']:
fname = target.deployment_data[key]
with open(fname, 'r') as fin:
with open('%s/bin/%s' % (mntdir, os.path.basename(fname)), 'w') as fout:
@@ -319,97 +533,13 @@
fout.write(fin.read())
os.fchmod(fout.fileno(), XMOD)
- def _bzr_info(self, url, bzrdir):
- cwd = os.getcwd()
- try:
- os.chdir('%s' % bzrdir)
- revno = subprocess.check_output(['bzr', 'revno']).strip()
- return {
- 'project_name': bzrdir,
- 'branch_vcs': 'bzr',
- 'branch_revision': revno,
- 'branch_url': url,
- }
- finally:
- os.chdir(cwd)
-
- def _git_info(self, url, gitdir):
- cwd = os.getcwd()
- try:
- os.chdir('%s' % gitdir)
- commit_id = subprocess.check_output(
- ['git', 'log', '-1', '--pretty=%H']).strip()
- return {
- 'project_name': url.rsplit('/')[-1],
- 'branch_vcs': 'git',
- 'branch_revision': commit_id,
- 'branch_url': url,
- }
- finally:
- os.chdir(cwd)
-
- def _create_repos(self, testdef, testdir):
- cwd = os.getcwd()
- try:
- os.chdir(testdir)
- for repo in testdef['install'].get('bzr-repos', []):
- logging.info("bzr branch %s" % repo)
- # Pass non-existent BZR_HOME value, or otherwise bzr may
- # have non-reproducible behavior because it may rely on
- # bzr whoami value, presence of ssh keys, etc.
- subprocess.check_call(['bzr', 'branch', repo],
- env={'BZR_HOME': '/dev/null', 'BZR_LOG': '/dev/null'})
- name = repo.replace('lp:', '').split('/')[-1]
- self._sw_sources.append(self._bzr_info(repo, name))
- for repo in testdef['install'].get('git-repos', []):
- logging.info("git clone %s" % repo)
- subprocess.check_call(['git', 'clone', repo])
- name = os.path.splitext(os.path.basename(repo))[0]
- self._sw_sources.append(self._git_info(repo, name))
- finally:
- os.chdir(cwd)
-
- def _create_target_install(self, testdef, hostdir, targetdir):
- with open('%s/install.sh' % hostdir, 'w') as f:
- f.write('set -ex\n')
- f.write('cd %s\n' % targetdir)
-
- # TODO how should we handle this for Android?
- if 'deps' in testdef['install'] and \
- testdef['install']['deps'] is not None:
- f.write('sudo apt-get update\n')
- f.write('sudo apt-get install -y ')
- for dep in testdef['install']['deps']:
- f.write('%s ' % dep)
- f.write('\n')
-
- if 'steps' in testdef['install'] and \
- testdef['install']['steps'] is not None:
- for cmd in testdef['install']['steps']:
- f.write('%s\n' % cmd)
-
- def _copy_test(self, hostdir, targetdir, testdef, testdef_repo=None):
- self._sw_sources = []
- utils.ensure_directory(hostdir)
- with open('%s/testdef.yaml' % hostdir, 'w') as f:
- f.write(yaml.dump(testdef))
-
- if 'install' in testdef:
- self._create_repos(testdef, hostdir)
- self._create_target_install(testdef, hostdir, targetdir)
-
- with open('%s/run.sh' % hostdir, 'w') as f:
- f.write('set -e\n')
- f.write('cd %s\n' % targetdir)
- if 'steps' in testdef['run'] \
- and testdef['run']['steps'] is not None:
- for cmd in testdef['run']['steps']:
- f.write('%s\n' % cmd)
-
- if testdef_repo:
- for filepath in glob(os.path.join(testdef_repo, '*')):
- shutil.copy2(filepath, hostdir)
- logging.info('copied all test files')
+ tc = target.deployment_data['lava_test_case']
+ with open(tc, 'r') as fin:
+ with open('%s/bin/lava-test-case' % mntdir, 'w') as fout:
+ fout.write('#!%s\n\n' % shcmd)
+ fout.write('ACK_FIFO=%s\n' % ACK_FIFO)
+ fout.write(fin.read())
+ os.fchmod(fout.fileno(), XMOD)
def _mk_runner_dirs(self, mntdir):
utils.ensure_directory('%s/bin' % mntdir)
@@ -424,34 +554,37 @@
with target.file_system(results_part, 'lava') as d:
self._mk_runner_dirs(d)
self._copy_runner(d, target)
- testdirs = []
+
+ testdef_loader = TestDefinitionLoader(self.context, target.scratch_dir)
if testdef_urls:
for url in testdef_urls:
- tdir = self._get_test_definition(d,
- ldir,
- url,
- target.scratch_dir,
- isrepo=False)
- testdirs.append(tdir)
+ testdef_loader.load_from_url(url)
if testdef_repos:
for repo in testdef_repos:
- tdir = self._get_test_definition(d,
- ldir,
- repo,
- target.scratch_dir,
- isrepo=True)
- testdirs.append(tdir)
+ testdef_loader.load_from_repo(repo)
+
+ tdirs = []
+ for testdef in testdef_loader.testdefs:
+ # android mount the partition under /system, while ubuntu
+ # mounts under /, so we have hdir for where it is on the
+ # host and tdir for how the target will see the path
+ hdir = '%s/tests/%s' % (d, testdef.test_run_id)
+ tdir = '%s/tests/%s' % (ldir, testdef.test_run_id)
+ testdef.copy_test(hdir, tdir)
+ tdirs.append(tdir)
with open('%s/lava-test-runner.conf' % d, 'w') as f:
- for testdir in testdirs:
+ for testdir in tdirs:
f.write('%s\n' % testdir)
with target.file_system(target.config.root_part, 'etc') as d:
target.deployment_data['lava_test_configure_startup'](d)
- def _bundle_results(self, target):
+ return testdef_loader.testdefs_by_uuid
+
+ def _bundle_results(self, target, signal_director, testdefs_by_uuid):
""" Pulls the results from the target device and builds a bundle
"""
results_part = target.deployment_data['lava_test_results_part_attr']
@@ -459,13 +592,15 @@
rdir = self.context.host_result_dir
with target.file_system(results_part, 'lava/results') as d:
- bundle = lava_test_shell.get_bundle(d, self._sw_sources)
+ bundle = lava_test_shell.get_bundle(d, testdefs_by_uuid)
utils.ensure_directory_empty(d)
- (fd, name) = tempfile.mkstemp(
- prefix='lava-test-shell', suffix='.bundle', dir=rdir)
- with os.fdopen(fd, 'w') as f:
- DocumentIO.dump(f, bundle)
+ signal_director.postprocess_bundle(bundle)
+
+ (fd, name) = tempfile.mkstemp(
+ prefix='lava-test-shell', suffix='.bundle', dir=rdir)
+ with os.fdopen(fd, 'w') as f:
+ DocumentIO.dump(f, bundle)
def _assert_target(self, target):
""" Ensure the target has the proper deployment data required by this
=== modified file 'lava_dispatcher/lava_test_shell.py'
@@ -18,6 +18,13 @@
# along
# with this program; if not, see <http://www.gnu.org/licenses>.
+"""
+Import test results from disk.
+
+This module contains functions to create a bundle from the disk files created
+by a lava-test-shell run.
+"""
+
import datetime
import decimal
import mimetypes
@@ -26,8 +33,6 @@
import os
import re
-from uuid import uuid4
-
from lava_dispatcher.test_data import create_attachment
@@ -250,20 +255,22 @@
return attachments
-def _get_test_run(test_run_dir, hwcontext, swcontext):
+def _get_test_run(test_run_dir, hwcontext, build, pkginfo, testdefs_by_uuid):
now = datetime.datetime.utcnow().strftime('%Y-%m-%dT%H:%M:%SZ')
testdef = _read_content(os.path.join(test_run_dir, 'testdef.yaml'))
stdout = _read_content(os.path.join(test_run_dir, 'stdout.log'))
+ uuid = _read_content(os.path.join(test_run_dir, 'analyzer_assigned_uuid'))
attachments = _get_run_attachments(test_run_dir, testdef, stdout)
attributes = _attributes_from_dir(os.path.join(test_run_dir, 'attributes'))
testdef = yaml.load(testdef)
+ swcontext = _get_sw_context(build, pkginfo, testdefs_by_uuid[uuid]._sw_sources)
return {
'test_id': testdef.get('metadata').get('name'),
'analyzer_assigned_date': now,
- 'analyzer_assigned_uuid': str(uuid4()),
+ 'analyzer_assigned_uuid': uuid,
'time_check_performed': False,
'test_results': _get_test_results(test_run_dir, testdef, stdout),
'software_context': swcontext,
@@ -287,7 +294,7 @@
for filename in os.listdir(dirpath)]
-def get_bundle(results_dir, sw_sources):
+def get_bundle(results_dir, testdefs_by_uuid):
"""
iterates through a results directory to build up a bundle formatted for
the LAVA dashboard
@@ -299,14 +306,13 @@
build = _read_content(os.path.join(results_dir, 'swcontext/build.txt'))
pkginfo = _read_content(os.path.join(results_dir, 'swcontext/pkgs.txt'), ignore_missing=True)
- swctx = _get_sw_context(build, pkginfo, sw_sources)
for test_run_name, test_run_path in _directory_names_and_paths(results_dir):
if test_run_name in ('hwcontext', 'swcontext'):
continue
if os.path.isdir(test_run_path):
try:
- testruns.append(_get_test_run(test_run_path, hwctx, swctx))
+ testruns.append(_get_test_run(test_run_path, hwctx, build, pkginfo, testdefs_by_uuid))
except:
logging.exception('error processing results for: %s' % test_run_name)
=== added directory 'lava_dispatcher/signals'
=== added file 'lava_dispatcher/signals/__init__.py'
@@ -0,0 +1,156 @@
+#!/usr/bin/python
+
+# Copyright (C) 2012 Linaro Limited
+#
+# Author: Andy Doan <andy.doan@linaro.org>
+#
+# This file is part of LAVA Dispatcher.
+#
+# LAVA Dispatcher is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# LAVA Dispatcher is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along
+# with this program; if not, see <http://www.gnu.org/licenses>.
+
+import logging
+
+
+class BaseSignalHandler(object):
+
+ def __init__(self, testdef_obj):
+ self.testdef_obj = testdef_obj
+
+ def start(self):
+ pass
+
+ def end(self):
+ pass
+
+ def starttc(self, test_case_id):
+ pass
+
+ def endtc(self, test_case_id):
+ pass
+
+ def custom_signal(self, signame, params):
+ pass
+
+ def postprocess_test_run(self, test_run):
+ pass
+
+
+class SignalHandler(BaseSignalHandler):
+
+ def __init__(self, testdef_obj):
+ BaseSignalHandler.__init__(self, testdef_obj)
+ self._case_data = {}
+ self._cur_case_id = None
+ self._cur_case_data = None
+
+ def starttc(self, test_case_id):
+ if self._cur_case_data:
+ logging.warning(
+ "unexpected cur_case_data %s", self._cur_case_data)
+ self._cur_case_id = test_case_id
+ data = None
+ try:
+ data = self.start_testcase(test_case_id)
+ except:
+ logging.exception("start_testcase failed for %s", test_case_id)
+ self._cur_case_data = self._case_data[test_case_id] = data
+
+ def endtc(self, test_case_id):
+ if self._cur_case_id != test_case_id:
+ logging.warning(
+ "stoptc for %s received but expecting %s",
+ test_case_id, self._cur_case_id)
+ else:
+ try:
+ self.end_testcase(test_case_id, self._cur_case_data)
+ except:
+ logging.exception(
+ "stop_testcase failed for %s", test_case_id)
+ self._cur_case_data = None
+
+ def postprocess_test_run(self, test_run):
+ for test_result in test_run['test_results']:
+ tc_id = test_result.get('test_case_id')
+ if not tc_id:
+ continue
+ if tc_id not in self._case_data:
+ continue
+ data = self._case_data[tc_id]
+ try:
+ self.postprocess_test_result(test_result, data)
+ except:
+ logging.exception("postprocess_test_result failed for %s", tc_id)
+
+ def start_testcase(self, test_case_id):
+ return {}
+
+ def end_testcase(self, test_case_id, data):
+ pass
+
+ def postprocess_test_result(self, test_result, case_data):
+ pass
+
+
+
+class SignalDirector(object):
+
+ def __init__(self, client, testdefs_by_uuid):
+ self.client = client
+ self.testdefs_by_uuid = testdefs_by_uuid
+ self._test_run_data = []
+ self._cur_handler = None
+
+ def signal(self, name, params):
+ handler = getattr(self, '_on_' + name, None)
+ if not handler and self._cur_handler:
+ handler = self._cur_handler.custom_signal
+ params = [name] + list(params)
+ if handler:
+ try:
+ handler(*params)
+ except:
+ logging.exception("handling signal %s failed", name)
+
+ def _on_STARTRUN(self, test_run_id, uuid):
+ self._cur_handler = None
+ testdef_obj = self.testdefs_by_uuid.get(uuid)
+ if testdef_obj:
+ self._cur_handler = testdef_obj.handler
+ if self._cur_handler:
+ self._cur_handler.start()
+
+ def _on_ENDRUN(self, test_run_id, uuid):
+ if self._cur_handler:
+ self._cur_handler.end()
+
+ def _on_STARTTC(self, test_case_id):
+ if self._cur_handler:
+ self._cur_handler.starttc(test_case_id)
+
+ def _on_ENDTC(self, test_case_id):
+ if self._cur_handler:
+ self._cur_handler.endtc(test_case_id)
+
+ def postprocess_bundle(self, bundle):
+ for test_run in bundle['test_runs']:
+ uuid = test_run['analyzer_assigned_uuid']
+ testdef_obj = self.testdefs_by_uuid.get(uuid)
+ if testdef_obj.handler:
+ try:
+ testdef_obj.handler.postprocess_test_run(test_run)
+ except:
+ logging.exception(
+ "postprocessing test run with uuid %s failed", uuid)
+
=== added file 'lava_dispatcher/signals/duration.py'
@@ -0,0 +1,20 @@
+import datetime
+import time
+
+from json_schema_validator.extensions import timedelta_extension
+
+from lava_dispatcher.signals import SignalHandler
+
+class AddDuration(SignalHandler):
+
+ def start_testcase(self, test_case_id):
+ return {
+ 'starttime': time.time()
+ }
+
+ def end_testcase(self, test_case_id, data):
+ data['endtime'] = time.time()
+
+ def postprocess_test_result(self, test_result, data):
+ delta = datetime.timedelta(seconds=data['endtime'] - data['starttime'])
+ test_result['duration'] = timedelta_extension.to_json(delta)
=== modified file 'lava_dispatcher/utils.py'
@@ -163,9 +163,9 @@
class logging_spawn(pexpect.spawn):
- def sendline(self, *args, **kw):
- logging.debug("sendline : %s" % args[0])
- return super(logging_spawn, self).sendline(*args, **kw)
+ def sendline(self, s=''):
+ logging.debug("sendline : %s" % s)
+ return super(logging_spawn, self).sendline(s)
def send(self, *args, **kw):
logging.debug("send : %s" % args[0])
=== modified file 'lava_test_shell/lava-test-case'
@@ -17,8 +17,13 @@
fi
if [ "$1" = "--shell" ]; then
shift
+ echo "<LAVA_SIGNAL_STARTTC $TEST_CASE_ID>"
+ read -t 600 < $ACK_FIFO
$*
- if [ $? -eq 0 ]; then
+ rc=$?
+ echo "<LAVA_SIGNAL_ENDTC $TEST_CASE_ID>"
+ read -t 600 < $ACK_FIFO
+ if [ $rc -eq 0 ]; then
RESULT=pass
else
RESULT=fail
=== modified file 'lava_test_shell/lava-test-runner-android'
@@ -12,6 +12,11 @@
# make sure we log to serial console
exec >/dev/console
+# This is a total hack to make sure we wait until the shell prompt has
+# appeared before sending any signals.
+sleep 15
+
+
PREFIX="<LAVA_TEST_RUNNER>:"
WORKFILE="/data/lava/lava-test-runner.conf"
RESULTSDIR="/data/lava/results"
@@ -94,6 +99,7 @@
mkdir ${odir}
mkdir ${odir}/attachments/
cp ${line}/testdef.yaml ${odir}/
+ cp ${line}/uuid ${odir}/analyzer_assigned_uuid
cp ${line}/run.sh ${odir}/attachments/
[ -f ${line}/install.sh ] && cp ${line}/install.sh ${odir}/attachments/
lava-test-shell --output_dir ${odir} /system/bin/sh -e "${line}/run.sh"
=== modified file 'lava_test_shell/lava-test-runner-ubuntu'
@@ -1,8 +1,12 @@
-#!/bin/sh
+#!/bin/bash
# make sure we log to serial console
exec >/dev/console
+# This is a total hack to make sure we wait until the shell prompt has
+# appeared before sending any signals.
+sleep 15
+
PREFIX="<LAVA_TEST_RUNNER>:"
WORKFILE="/lava/lava-test-runner.conf"
RESULTSDIR="/lava/results"
@@ -77,10 +81,13 @@
odir=${RESULTSDIR}/${test}-`date +%s`
mkdir ${odir}
mkdir ${odir}/attachments/
+ cp ${line}/uuid ${odir}/analyzer_assigned_uuid
cp ${line}/testdef.yaml ${odir}/
cp ${line}/run.sh ${odir}/attachments/
[ -f ${line}/install.sh ] && cp ${line}/install.sh ${odir}/attachments/
- lava-test-shell --output_dir ${odir} /bin/sh -e "${line}/run.sh"
+ # run.sh includes a "read -t <timeout>" which isn't supported by dash
+ # so be sure to use bash
+ lava-test-shell --output_dir ${odir} /bin/bash -e "${line}/run.sh"
echo "${PREFIX} ${test} exited with: `cat ${odir}/return_code`"
done < ${WORKFILE}
=== modified file 'setup.py'
@@ -16,6 +16,8 @@
dispatch = lava.dispatcher.commands:dispatch
connect = lava.dispatcher.commands:connect
power-cycle = lava.dispatcher.commands:power_cycle
+ [lava.signal_handlers]
+ add-duration = lava_dispatcher.signals.duration:AddDuration
""",
packages=find_packages(),
package_data= {
@@ -33,6 +35,7 @@
'lava_test_shell/lava-test-runner-android',
'lava_test_shell/lava-test-runner-ubuntu',
'lava_test_shell/lava-test-runner.conf',
+ 'lava_test_shell/lava-test-case',
'lava_test_shell/lava-test-runner.init.d',
'lava_test_shell/lava-test-shell',
])