From patchwork Wed Sep 26 22:16:11 2012 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael-Doyle Hudson X-Patchwork-Id: 11746 Return-Path: X-Original-To: patchwork@peony.canonical.com Delivered-To: patchwork@peony.canonical.com Received: from fiordland.canonical.com (fiordland.canonical.com [91.189.94.145]) by peony.canonical.com (Postfix) with ESMTP id 6628D23EB4 for ; Wed, 26 Sep 2012 22:16:16 +0000 (UTC) Received: from mail-ie0-f180.google.com (mail-ie0-f180.google.com [209.85.223.180]) by fiordland.canonical.com (Postfix) with ESMTP id AF567A18218 for ; Wed, 26 Sep 2012 22:16:15 +0000 (UTC) Received: by ieje10 with SMTP id e10so2699842iej.11 for ; Wed, 26 Sep 2012 15:16:15 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=x-forwarded-to:x-forwarded-for:delivered-to:received-spf :content-type:mime-version:x-launchpad-project:x-launchpad-branch :x-launchpad-message-rationale:x-launchpad-branch-revision-number :x-launchpad-notification-type:to:from:subject:message-id:date :reply-to:sender:errors-to:precedence:x-generated-by :x-launchpad-hash:x-gm-message-state; bh=NqQYG+gRyCJW52q5I2ZDL9QrS39yW64B/K4Ki2fn63U=; b=B7jDPeovxvPzbMHQ6pIbRzAOk0/1OOlfqQt7lwZWV2syoR+j20ixQhlUO1+2dRioQ3 AbMRRDGxk+KGoh6D6pr5CqzUnC7CzEuTDEhXLrsAD6FXQC6MglIxZjrEEY4S8SUejHXL Jr3Jbo08oRqIo6n2GAXiA40dfl69fnGv1TyQAzqHLzDpXC5UDjIyLNr8cPodpVQaS1mV gB/05469AUeG1CkcRWfZfh/FaQZFzON6kbJ5jeadTnQ6+pfIUfeHehDIMvGzHLlDHuvz cfdxIDC7Za3gWhHo4Na7K5Wf4bTyphM7hJesNv7lIy9WLf3O+KjL42AiLo5ihpxt1x37 BbnA== Received: by 10.50.217.227 with SMTP id pb3mr12518333igc.28.1348697775066; Wed, 26 Sep 2012 15:16:15 -0700 (PDT) X-Forwarded-To: linaro-patchwork@canonical.com X-Forwarded-For: patch@linaro.org linaro-patchwork@canonical.com Delivered-To: patches@linaro.org Received: by 10.50.184.232 with SMTP id ex8csp382538igc; Wed, 26 Sep 2012 15:16:13 -0700 (PDT) Received: by 10.216.197.104 with SMTP id s82mr335600wen.62.1348697772738; Wed, 26 Sep 2012 15:16:12 -0700 (PDT) Received: from indium.canonical.com (indium.canonical.com. [91.189.90.7]) by mx.google.com with ESMTPS id fe9si11115736wib.28.2012.09.26.15.16.12 (version=TLSv1/SSLv3 cipher=OTHER); Wed, 26 Sep 2012 15:16:12 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of bounces@canonical.com designates 91.189.90.7 as permitted sender) client-ip=91.189.90.7; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of bounces@canonical.com designates 91.189.90.7 as permitted sender) smtp.mail=bounces@canonical.com Received: from ackee.canonical.com ([91.189.89.26]) by indium.canonical.com with esmtp (Exim 4.71 #1 (Debian)) id 1TGztr-0004L6-WA for ; Wed, 26 Sep 2012 22:16:12 +0000 Received: from ackee.canonical.com (localhost [127.0.0.1]) by ackee.canonical.com (Postfix) with ESMTP id E329FE012B for ; Wed, 26 Sep 2012 22:16:11 +0000 (UTC) MIME-Version: 1.0 X-Launchpad-Project: lava-dispatcher X-Launchpad-Branch: ~linaro-validation/lava-dispatcher/trunk X-Launchpad-Message-Rationale: Subscriber X-Launchpad-Branch-Revision-Number: 390 X-Launchpad-Notification-Type: branch-revision To: Linaro Patch Tracker From: noreply@launchpad.net Subject: [Branch ~linaro-validation/lava-dispatcher/trunk] Rev 390: use configglue to define a schema for our configuration Message-Id: <20120926221611.16055.36482.launchpad@ackee.canonical.com> Date: Wed, 26 Sep 2012 22:16:11 -0000 Reply-To: noreply@launchpad.net Sender: bounces@canonical.com Errors-To: bounces@canonical.com Precedence: bulk X-Generated-By: Launchpad (canonical.com); Revision="16033"; Instance="launchpad-lazr.conf" X-Launchpad-Hash: 383c2b4cad95ca05618af564644acfcdca236a80 X-Gm-Message-State: ALoCoQlqBd8h8BDFTxhoo0k7JLUCYI4KJu56UnsBWPNLqMpNLJXofaATxHBtoN8rT/P/20zu7j/s Merge authors: Michael Hudson-Doyle (mwhudson) Related merge proposals: https://code.launchpad.net/~mwhudson/lava-dispatcher/use-configglue/+merge/126370 proposed by: Michael Hudson-Doyle (mwhudson) review: Approve - Andy Doan (doanac) ------------------------------------------------------------ revno: 390 [merge] committer: Michael Hudson-Doyle branch nick: trunk timestamp: Thu 2012-09-27 10:15:10 +1200 message: use configglue to define a schema for our configuration buyer beware: this may be a little disruptive, test before upgrading! modified: lava/dispatcher/commands.py lava_dispatcher/actions/android_install_binaries.py lava_dispatcher/actions/lava-test.py lava_dispatcher/client/base.py lava_dispatcher/client/fastmodel.py lava_dispatcher/client/lmc_utils.py lava_dispatcher/client/master.py lava_dispatcher/client/qemu.py lava_dispatcher/config.py lava_dispatcher/context.py lava_dispatcher/default-config/lava-dispatcher/device-defaults.conf lava_dispatcher/tests/test_config.py setup.py --- lp:lava-dispatcher https://code.launchpad.net/~linaro-validation/lava-dispatcher/trunk You are subscribed to branch lp:lava-dispatcher. To unsubscribe from this branch go to https://code.launchpad.net/~linaro-validation/lava-dispatcher/trunk/+edit-subscription === modified file 'lava/dispatcher/commands.py' --- lava/dispatcher/commands.py 2012-06-14 03:46:32 +0000 +++ lava/dispatcher/commands.py 2012-09-26 02:55:25 +0000 @@ -70,9 +70,8 @@ FORMAT = '%(asctime)s %(levelname)s: %(message)s' DATEFMT= '%Y-%m-%d %I:%M:%S %p' logging.basicConfig(format=FORMAT,datefmt=DATEFMT) - config = get_config("lava-dispatcher", self.args.config_dir) - logging_level = config.get("LOGGING_LEVEL") - logging.root.setLevel(int(logging_level)) + config = get_config(self.args.config_dir) + logging.root.setLevel(config.logging_level) # Set process id if job-id was passed to dispatcher if self.args.job_id: @@ -120,12 +119,12 @@ def invoke(self): os.execlp( - 'sh', 'sh', '-c', self.device_config.get('connection_command')) + 'sh', 'sh', '-c', self.device_config.connection_command) class power_cycle(DeviceCommand): def invoke(self): - command = self.device_config.get('hard_reset_command', '') + command = self.device_config.hard_reset_command if not command: raise CommandError( "%s does not have a power cycle command configured" % === modified file 'lava_dispatcher/actions/android_install_binaries.py' --- lava_dispatcher/actions/android_install_binaries.py 2012-03-11 22:12:52 +0000 +++ lava_dispatcher/actions/android_install_binaries.py 2012-09-26 02:41:46 +0000 @@ -17,7 +17,6 @@ # You should have received a copy of the GNU General Public License # along with this program; if not, see . -import ConfigParser import logging from lava_dispatcher.actions import BaseAction, null_or_empty_schema from lava_dispatcher.client.master import _deploy_tarball_to_board @@ -28,10 +27,8 @@ parameters_schema = null_or_empty_schema def run(self): - try: - driver_tarball = self.client.device_option( - "android_binary_drivers") - except ConfigParser.NoOptionError: + driver_tarball = self.client.config.android_binary_drivers + if driver_tarball is None: logging.error("android_binary_drivers not defined in any config") return === modified file 'lava_dispatcher/actions/lava-test.py' --- lava_dispatcher/actions/lava-test.py 2012-08-01 15:41:23 +0000 +++ lava_dispatcher/actions/lava-test.py 2012-09-26 02:55:25 +0000 @@ -39,13 +39,13 @@ dispatcher_config = client.context.config - lava_test_deb = dispatcher_config.get("LAVA_TEST_DEB", "") - if lava_test_deb != "": + lava_test_deb = dispatcher_config.lava_test_deb + if lava_test_deb: logging.debug("Installing %s with apt-get" % lava_test_deb) session.run("%s -y --force-yes install %s" % (client.aptget_cmd, lava_test_deb)) else: - lava_test_url = dispatcher_config.get("LAVA_TEST_URL") + lava_test_url = dispatcher_config.lava_test_url logging.debug("Installing %s with pip" % lava_test_url) session.run('pip install -e ' + lava_test_url) === modified file 'lava_dispatcher/client/base.py' --- lava_dispatcher/client/base.py 2012-09-12 20:16:12 +0000 +++ lava_dispatcher/client/base.py 2012-09-26 02:55:25 +0000 @@ -127,7 +127,7 @@ def _check_network_up(self): """Internal function for checking network once.""" - lava_server_ip = self._client.context.lava_server_ip + lava_server_ip = self._client.context.config.lava_server_ip self.run( "LC_ALL=C ping -W4 -c1 %s" % lava_server_ip, ["1 received", "0 received", "Network is unreachable"], @@ -154,7 +154,7 @@ def __init__(self, client, wait_for_rc=True): CommandRunner.__init__( - self, client.proc, client.tester_str, wait_for_rc) + self, client.proc, client.config.tester_str, wait_for_rc) def export_display(self): self.run("su - linaro -c 'DISPLAY=:0 xhost local:'", failok=True) @@ -169,7 +169,7 @@ def __init__(self, client): super(AndroidTesterCommandRunner, self).__init__( - client, client.tester_str, wait_for_rc=False) + client, client.config.tester_str, wait_for_rc=False) self.dev_name = None # adb cound be connected through network @@ -274,54 +274,12 @@ def __init__(self, context, config): self.context = context self.config = config + self.hostname = config.hostname self.sio = SerialIO(sys.stdout) self.proc = None # used for apt-get in lava-test.py self.aptget_cmd = "apt-get" - def device_option(self, option_name, *extra): - return self.config.get(option_name, *extra) - - def device_option_int(self, option_name): - return self.config.getint(option_name) - - @property - def hostname(self): - return self.device_option("hostname") - - @property - def tester_hostname(self): - return self.device_option("tester_hostname", "linaro") - - @property - def tester_str(self): - return self.device_option("TESTER_STR") - - @property - def device_type(self): - return self.device_option("device_type") - - @property - def boot_part(self): - return self.device_option_int("boot_part") - - @property - def root_part(self): - return self.device_option_int("root_part") - - @property - def default_network_interface(self): - return self.device_option("default_network_interface") - - @property - def lmc_dev_arg(self): - return self.device_option("lmc_dev_arg") - - @property - def enable_network_after_boot_android(self): - return self.config.getboolean( - 'enable_network_after_boot_android', True) - @contextlib.contextmanager def tester_session(self): """A session that can be used to run commands booted into the test @@ -387,13 +345,13 @@ if self.proc is None: raise OperationFailed self.proc.sendline("") - match_id = self.proc.expect([self.tester_str, pexpect.TIMEOUT], + match_id = self.proc.expect([self.config.tester_str, pexpect.TIMEOUT], timeout=timeout) if match_id == 1: raise OperationFailed def setup_proxy(self, prompt_str): - lava_proxy = self.context.lava_proxy + lava_proxy = self.context.config.lava_proxy if lava_proxy: logging.info("Setting up http proxy") # haven't included Android support yet @@ -415,23 +373,23 @@ logging.info("Boot the test image") self._boot_linaro_image() - timeout = self.config.getint("boot_linaro_timeout", 300) + timeout = self.config.boot_linaro_timeout self.in_test_shell(timeout) # set PS1 to include return value of last command # Details: system PS1 is set in /etc/bash.bashrc and user PS1 is set in # /root/.bashrc, it is # "${debian_chroot:+($debian_chroot)}\u@\h:\w\$ " self.proc.sendline('export PS1="$PS1 [rc=$(echo \$?)]: "') - self.proc.expect(self.tester_str, timeout=120) + self.proc.expect(self.config.tester_str, timeout=120) - self.setup_proxy(self.tester_str) + self.setup_proxy(self.config.tester_str) logging.info("System is in test image now") def get_www_scratch_dir(self): - ''' returns a temporary directory available for downloads that's gets - deleted when the process exits ''' + """returns a temporary directory available for downloads that's gets + deleted when the process exits""" - d = mkdtemp(dir=self.context.lava_image_tmpdir) + d = mkdtemp(dir=self.context.config.lava_image_tmpdir) atexit.register(shutil.rmtree, d) os.chmod(d, 0755) return d @@ -443,21 +401,21 @@ # Android stuff def get_android_adb_interface(self): - return self.default_network_interface + return self.config.default_network_interface def boot_linaro_android_image(self): """Reboot the system to the test android image.""" self._boot_linaro_android_image() self.in_test_shell(timeout=900) self.proc.sendline("export PS1=\"root@linaro: \"") - self.proc.expect(self.tester_str, timeout=120) + self.proc.expect(self.config.tester_str, timeout=120) #TODO: set up proxy # we are tcp'ish adb fans here... self._disable_adb_over_usb() self._disable_suspend() - if self.enable_network_after_boot_android: + if self.config.enable_network_after_boot_android: time.sleep(1) self._enable_network() @@ -477,15 +435,16 @@ else: logging.info("Skip raising exception on the home screen has not displayed for health check jobs") - timeout = self.config.getint("disablesuspend_timeout", 240) - session.run('/system/bin/disablesuspend.sh', timeout=timeout) + session.run( + '/system/bin/disablesuspend.sh', + timeout=self.config.disablesuspend_timeout) def _enable_network(self): session = TesterCommandRunner(self, wait_for_rc=False) session.run("netcfg", timeout=20) - session.run("netcfg %s up" % self.default_network_interface, timeout=20) - session.run("netcfg %s dhcp" % self.default_network_interface, timeout=300) - session.run("ifconfig " + self.default_network_interface, timeout=20) + session.run("netcfg %s up" % self.config.default_network_interface, timeout=20) + session.run("netcfg %s dhcp" % self.config.default_network_interface, timeout=300) + session.run("ifconfig " + self.config.default_network_interface, timeout=20) def _restart_adb_after_netup(self): === modified file 'lava_dispatcher/client/fastmodel.py' --- lava_dispatcher/client/fastmodel.py 2012-09-06 20:41:35 +0000 +++ lava_dispatcher/client/fastmodel.py 2012-09-26 02:41:46 +0000 @@ -20,17 +20,16 @@ import atexit import codecs -import contextlib import cStringIO import logging import os -import pexpect import shutil import stat import threading import time from lava_dispatcher.client.base import ( + CriticalError, TesterCommandRunner, LavaClient, ) @@ -38,7 +37,6 @@ image_partition_mounted, generate_android_image, generate_fastmodel_image, - get_partition_offset, ) from lava_dispatcher.downloader import ( download_image, @@ -71,10 +69,8 @@ def __init__(self, context, config): super(LavaFastModelClient, self).__init__(context, config) - self._sim_binary = config.get('simulator_binary', None) - lic_server = config.get('license_server', None) - self.git_url_disablesuspend_sh = config.get('git_url_disablesuspend_sh', - None) + self._sim_binary = config.simulator_binary + lic_server = config.license_server if not self._sim_binary or not lic_server: raise RuntimeError("The device type config for this device " "requires settings for 'simulator_binary' and 'license_server'") @@ -93,24 +89,24 @@ with image_partition_mounted(self._sd_image, self.SYS_PARTITION) as d: script_path = '%s/%s' % (d, 'bin/disablesuspend.sh') - if self.git_url_disablesuspend_sh: + if self.config.git_url_disablesuspend_sh: logging_system('sudo wget %s -O %s' % ( - self.git_url_disablesuspend_sh, + self.config.git_url_disablesuspend_sh, script_path)) logging_system('sudo chmod +x %s' % script_path) logging_system('sudo chown :2000 %s' % script_path) #make sure PS1 is what we expect it to be logging_system( - 'sudo sh -c \'echo "PS1=%s: ">> %s/etc/mkshrc\'' % (self.tester_str, d)) + 'sudo sh -c \'echo "PS1=%s: ">> %s/etc/mkshrc\'' % (self.config.tester_str, d)) # fast model usermode networking does not support ping logging_system( 'sudo sh -c \'echo "alias ping=\\\"echo LAVA-ping override 1 received\\\"">> %s/etc/mkshrc\'' % d) def _customize_ubuntu(self): - with image_partition_mounted(self._sd_image, self.root_part) as mntdir: + with image_partition_mounted(self._sd_image, self.config.root_part) as mntdir: logging_system('sudo echo %s > %s/etc/hostname' - % (self.tester_hostname, mntdir)) + % (self.config.tester_hostname, mntdir)) def deploy_image(self, image, axf, is_android=False): self._axf = download_image(axf, self.context) @@ -141,7 +137,7 @@ generate_android_image( 'vexpress-a9', self._boot, self._data, self._system, self._sd_image) - self._copy_axf(self.boot_part, 'linux-system-ISW.axf') + self._copy_axf(self.client.boot_part, 'linux-system-ISW.axf') self._customize_android() @@ -165,7 +161,7 @@ self._axf = '%s/img.axf' % odir else: self._sd_image = download_image(image, self.context) - self._copy_axf(self.root_part, 'boot/img.axf') + self._copy_axf(self.config.root_part, 'boot/img.axf') self._customize_ubuntu() @@ -283,7 +279,7 @@ tardir = os.path.dirname(self._sd_image) tarfile = os.path.join(tardir, 'lava_results.tgz') - with image_partition_mounted(self._sd_image, self.root_part) as mnt: + with image_partition_mounted(self._sd_image, self.config.root_part) as mnt: logging_system( 'tar czf %s -C %s%s .' % ( tarfile, mnt, self.context.lava_result_dir)) === modified file 'lava_dispatcher/client/lmc_utils.py' --- lava_dispatcher/client/lmc_utils.py 2012-09-06 20:16:10 +0000 +++ lava_dispatcher/client/lmc_utils.py 2012-09-26 02:41:46 +0000 @@ -40,11 +40,11 @@ image_file = os.path.join(outdir, "lava.img") - logging.info("client.device_type = %s" %client.device_type) + logging.info("client.device_type = %s" %client.config.device_type) cmd = ("sudo flock /var/lock/lava-lmc.lck linaro-media-create --hwpack-force-yes --dev %s " "--image-file %s --binary %s --hwpack %s --image-size 3G" % - (client.lmc_dev_arg, image_file, rootfs_path, hwpack_path)) + (client.config.lmc_dev_arg, image_file, rootfs_path, hwpack_path)) if rootfstype is not None: cmd += ' --rootfs ' + rootfstype logging.info("Executing the linaro-media-create command") === modified file 'lava_dispatcher/client/master.py' --- lava_dispatcher/client/master.py 2012-09-20 21:34:50 +0000 +++ lava_dispatcher/client/master.py 2012-09-26 02:55:25 +0000 @@ -24,7 +24,6 @@ import os import re import shutil -import subprocess import time import traceback import atexit @@ -115,7 +114,7 @@ _deploy_tarball_to_board(session, rootfs, '/mnt/root', timeout=18000) session.run('echo %s > /mnt/root/etc/hostname' - % session._client.tester_hostname) + % session._client.config.tester_hostname) #DO NOT REMOVE - diverting flash-kernel and linking it to /bin/true #prevents a serious problem where packages getting installed that #call flash-kernel can update the kernel on the master image @@ -149,12 +148,12 @@ def _update_uInitrd_partitions(session, rc_filename): # Original android sdcard partition layout by l-a-m-c - sys_part_org = session._client.device_option("sys_part_android_org") - cache_part_org = session._client.device_option("cache_part_android_org") - data_part_org = session._client.device_option("data_part_android_org") + sys_part_org = session._client.config.sys_part_android_org + cache_part_org = session._client.config.cache_part_android_org + data_part_org = session._client.config.data_part_android_org # Sdcard layout in Lava image - sys_part_lava = session._client.device_option("sys_part_android") - data_part_lava = session._client.device_option("data_part_android") + sys_part_lava = session._client.config.sys_part_android + data_part_lava = session._client.config.data_part_android session.run( 'sed -i "/mount ext4 \/dev\/block\/mmcblk0p%s/d" %s' @@ -164,7 +163,6 @@ % (data_part_org, data_part_lava, rc_filename), failok=True) session.run('sed -i "s/mmcblk0p%s/mmcblk0p%s/g" %s' % (sys_part_org, sys_part_lava, rc_filename), failok=True) - # for snowball the mcvblk1 is used instead of mmcblk0. session.run('sed -i "s/mmcblk1p%s/mmcblk1p%s/g" %s' % (data_part_org, data_part_lava, rc_filename), failok=True) session.run('sed -i "s/mmcblk1p%s/mmcblk1p%s/g" %s' @@ -223,9 +221,8 @@ # If there is no userdata partition on the sdcard(like iMX and Origen), # then the sdcard partition will be used as the userdata partition as # before, and so cannot be used here as the sdcard on android - sdcard_part_lava = session._client.device_option("sdcard_part_android") - sdcard_part_org = session._client.device_option( - "sdcard_part_android_org") + sdcard_part_lava = session._client.config.sdcard_part_android + sdcard_part_org = session._client.config.sdcard_part_android_org original = 'dev_mount sdcard /mnt/sdcard %s ' % sdcard_part_org replacement = 'dev_mount sdcard /mnt/sdcard %s ' % sdcard_part_lava sed_cmd = "s@{original}@{replacement}@".format(original=original, @@ -237,8 +234,8 @@ script_path = '%s/%s' % ('/mnt/lava', '/system/bin/disablesuspend.sh') if not session.is_file_exist(script_path): - git_url = session._client.device_option("git_url_disablesuspend_sh") - lava_proxy = session._client.context.lava_proxy + git_url = session._client.config.git_url_disablesuspend_sh + lava_proxy = session._client.context.config.lava_proxy session.run("sh -c 'export http_proxy=%s'" % lava_proxy) session.run('wget --no-check-certificate %s -O %s' % (git_url, script_path)) session.run('chmod +x %s' % script_path) @@ -296,7 +293,7 @@ """ def __init__(self, client): - super(MasterCommandRunner, self).__init__(client, client.master_str) + super(MasterCommandRunner, self).__init__(client, client.config.master_str) def get_master_ip(self): #get master image ip address @@ -308,7 +305,7 @@ #tty device uses minimal match, see pexpect wiki pattern1 = "<(\d?\d?\d?\.\d?\d?\d?\.\d?\d?\d?\.\d?\d?\d?)>" cmd = ("ifconfig %s | grep 'inet addr' | awk -F: '{print $2}' |" - "awk '{print \"<\" $1 \">\"}'" % self._client.default_network_interface) + "awk '{print \"<\" $1 \">\"}'" % self._client.config.default_network_interface) self.run( cmd, [pattern1, pexpect.EOF, pexpect.TIMEOUT], timeout=5) if self.match_id == 0: @@ -336,14 +333,14 @@ def __init__(self, context, config): super(LavaMasterImageClient, self).__init__(context, config) - pre_connect = self.device_option("pre_connect_command") + pre_connect = self.config.pre_connect_command if pre_connect: logging_system(pre_connect) self.proc = self._connect_carefully() atexit.register(self._close_logging_spawn) def _connect_carefully(self): - cmd = self.device_option("connection_command") + cmd = self.config.connection_command retry_count = 0 retry_limit = 3 @@ -380,7 +377,7 @@ elif result == 'all-good': return proc elif result == 'reset-port': - reset_port = self.device_option("reset_port_command") + reset_port = self.config.reset_port_command if reset_port: logging_system(reset_port) else: @@ -390,10 +387,6 @@ time.sleep(5) raise OperationFailed("could execute connection_command successfully") - @property - def master_str(self): - return self.device_option("MASTER_STR") - def _close_logging_spawn(self): self.proc.close(True) @@ -475,8 +468,8 @@ shutil.copy(root_tgz, c_root_tgz) def deploy_linaro(self, hwpack=None, rootfs=None, image=None, rootfstype='ext3'): - LAVA_IMAGE_TMPDIR = self.context.lava_image_tmpdir - LAVA_IMAGE_URL = self.context.lava_image_url + LAVA_IMAGE_TMPDIR = self.context.config.lava_image_tmpdir + LAVA_IMAGE_URL = self.context.config.lava_image_url # validate in parameters if image is None: if hwpack is None or rootfs is None: @@ -495,7 +488,7 @@ boot_tgz, root_tgz = self._generate_tarballs(image_file) else: os.chmod(tarball_dir, 0755) - lava_cachedir = self.context.lava_cachedir + lava_cachedir = self.context.config.lava_cachedir if self.context.job_data.get('health_check', False): if self._are_tarballs_cached(image, lava_cachedir): logging.info("Reusing cached tarballs") @@ -635,18 +628,18 @@ logging.info("Boot the system master image") try: self.soft_reboot() - image_boot_msg = self.device_option('image_boot_msg') + image_boot_msg = self.config.image_boot_msg self.proc.expect(image_boot_msg, timeout=300) self._in_master_shell(300) except: logging.exception("in_master_shell failed") self.hard_reboot() - image_boot_msg = self.device_option('image_boot_msg') + image_boot_msg = self.config.image_boot_msg self.proc.expect(image_boot_msg, timeout=300) self._in_master_shell(300) self.proc.sendline('export PS1="$PS1 [rc=$(echo \$?)]: "') - self.proc.expect(self.master_str, timeout=120, lava_no_logging=1) - self.setup_proxy(self.master_str) + self.proc.expect(self.config.master_str, timeout=120, lava_no_logging=1) + self.setup_proxy(self.config.master_str) logging.info("System is in master image now") def _format_testpartition(self, session, fstype): @@ -668,8 +661,8 @@ boot_tgz = os.path.join(tarball_dir, "boot.tgz") root_tgz = os.path.join(tarball_dir, "root.tgz") try: - _extract_partition(image_file, self.boot_part, boot_tgz) - _extract_partition(image_file, self.root_part, root_tgz) + _extract_partition(image_file, self.config.boot_part, boot_tgz) + _extract_partition(image_file, self.config.root_part, root_tgz) except: logging.error("Failed to generate tarballs") shutil.rmtree(tarball_dir) @@ -687,23 +680,23 @@ session.run('mkdir -p /mnt/root') session.run( 'mount /dev/disk/by-label/%s /mnt/root' % result_disk) + lava_result_dir = self.context.config.lava_result_dir # Clean results directory on master image session.run( - 'rm -rf /tmp/lava_results.tgz /tmp/%s' % self.context.lava_result_dir) - session.run('mkdir -p /tmp/%s' % self.context.lava_result_dir) + 'rm -rf /tmp/lava_results.tgz /tmp/%s' % lava_result_dir) + session.run('mkdir -p /tmp/%s' % lava_result_dir) session.run( - 'cp /mnt/root/%s/*.bundle /tmp/%s' % (self.context.lava_result_dir, - self.context.lava_result_dir)) + 'cp /mnt/root/%s/*.bundle /tmp/%s' % (lava_result_dir, lava_result_dir)) # Clean result bundle on test image session.run( - 'rm -f /mnt/root/%s/*.bundle' % (self.context.lava_result_dir)) + 'rm -f /mnt/root/%s/*.bundle' % (lava_result_dir)) session.run('umount /mnt/root') # Create tarball of all results logging.info("Creating lava results tarball") session.run('cd /tmp') session.run( - 'tar czf /tmp/lava_results.tgz -C /tmp/%s .' % self.context.lava_result_dir) + 'tar czf /tmp/lava_results.tgz -C /tmp/%s .' % lava_result_dir) # start gather_result job, status err_msg = '' @@ -772,7 +765,7 @@ master_session.run('mount --rbind /dev %s/dev' % directory) try: yield PrefixCommandRunner( - 'chroot ' + directory, self.proc, self.master_str) + 'chroot ' + directory, self.proc, self.config.master_str) finally: master_session.run( '[ -e %s/etc/resolv.conf.bak ] && cp -f %s/etc/resolv.conf.bak %s/etc/resolv.conf || rm %s/etc/resolv.conf' % ( @@ -788,7 +781,7 @@ """ self.proc.sendline("") match_id = self.proc.expect( - [self.master_str, pexpect.TIMEOUT], timeout=timeout, lava_no_logging=1) + [self.config.master_str, pexpect.TIMEOUT], timeout=timeout, lava_no_logging=1) if match_id == 1: raise OperationFailed @@ -808,7 +801,7 @@ def soft_reboot(self): logging.info("Perform soft reboot the system") - cmd = self.device_option("soft_boot_cmd") + cmd = self.config.soft_boot_cmd # make sure in the shell (sometime the earlier command has not exit) by sending CTRL + C self.proc.sendline("\003") if cmd != "": @@ -825,7 +818,7 @@ def hard_reboot(self): logging.info("Perform hard reset on the system") - cmd = self.device_option("hard_reset_command", "") + cmd = self.config.hard_reset_command if cmd != "": logging_system(cmd) else: @@ -842,11 +835,11 @@ ['.+', pexpect.EOF, pexpect.TIMEOUT], timeout=1, lava_no_logging=1) def _enter_uboot(self): - interrupt_boot_prompt = self.device_option('interrupt_boot_prompt') + interrupt_boot_prompt = self.config.interrupt_boot_prompt if self.proc.expect(interrupt_boot_prompt) != 0: raise Exception("Faile to enter uboot") - interrupt_boot_command = self.device_option('interrupt_boot_command') + interrupt_boot_command = self.config.interrupt_boot_command self.proc.sendline(interrupt_boot_command) def _boot_linaro_image(self): @@ -860,10 +853,10 @@ else: boot_cmds = keyval[1].strip() - self._boot(string_to_list(self.config.get(boot_cmds))) + self._boot(string_to_list(getattr(self.config, boot_cmds))) def _boot_linaro_android_image(self): - self._boot(string_to_list(self.config.get('boot_cmds_android'))) + self._boot(string_to_list(self.config.boot_cmds_android)) def _boot(self, boot_cmds): try: @@ -874,7 +867,7 @@ self.hard_reboot() self._enter_uboot() self.proc.sendline(boot_cmds[0]) - bootloader_prompt = re.escape(self.device_option('bootloader_prompt')) + bootloader_prompt = re.escape(self.config.bootloader_prompt) for line in range(1, len(boot_cmds)): self.proc.expect(bootloader_prompt, timeout=300) self.proc.sendline(boot_cmds[line]) === modified file 'lava_dispatcher/client/qemu.py' --- lava_dispatcher/client/qemu.py 2012-09-06 20:16:10 +0000 +++ lava_dispatcher/client/qemu.py 2012-09-26 02:33:58 +0000 @@ -22,6 +22,7 @@ import logging import os import pexpect +from tempfile import mkdtemp from lava_dispatcher.client.base import ( CommandRunner, @@ -54,8 +55,8 @@ else: image_file = download_image(image, self.context) self._lava_image = image_file - with image_partition_mounted(self._lava_image, self.root_part) as mntdir: - logging_system('echo %s > %s/etc/hostname' % (self.tester_hostname, + with image_partition_mounted(self._lava_image, self.config.root_part) as mntdir: + logging_system('echo %s > %s/etc/hostname' % (self.config.tester_hostname, mntdir)) @contextlib.contextmanager @@ -74,7 +75,7 @@ @contextlib.contextmanager def _chroot_into_rootfs_session(self): - with image_partition_mounted(self._lava_image, self.root_part) as mntdir: + with image_partition_mounted(self._lava_image, self.config.root_part) as mntdir: with self._mnt_prepared_for_qemu(mntdir): cmd = pexpect.spawn('chroot ' + mntdir, logfile=self.sio, timeout=None) try: @@ -101,32 +102,32 @@ """ if self.proc is not None: self.proc.sendline('sync') - self.proc.expect([self.tester_str, pexpect.TIMEOUT], timeout=10) + self.proc.expect([self.config.tester_str, pexpect.TIMEOUT], timeout=10) self.proc.close() qemu_cmd = ('%s -M %s -drive if=%s,cache=writeback,file=%s ' '-clock unix -device usb-kbd -device usb-mouse -usb ' '-device usb-net,netdev=mynet -netdev user,id=mynet ' '-net nic -net user -nographic') % ( - self.context.config.get('default_qemu_binary'), - self.device_option('qemu_machine_type'), - self.device_option('qemu_drive_interface'), + self.context.config.default_qemu_binary, + self.config.qemu_machine_type, + self.config.qemu_drive_interface, self._lava_image) logging.info('launching qemu with command %r' % qemu_cmd) self.proc = logging_spawn( qemu_cmd, logfile=self.sio, timeout=None) - self.proc.expect(self.tester_str, timeout=300) + self.proc.expect(self.config.tester_str, timeout=300) # set PS1 to include return value of last command self.proc.sendline('export PS1="$PS1 [rc=$(echo \$?)]: "') - self.proc.expect(self.tester_str, timeout=10) + self.proc.expect(self.config.tester_str, timeout=10) def retrieve_results(self, result_disk): if self.proc is not None: self.proc.sendline('sync') - self.proc.expect([self.tester_str, pexpect.TIMEOUT], timeout=10) + self.proc.expect([self.config.tester_str, pexpect.TIMEOUT], timeout=10) self.proc.close() tardir = mkdtemp() tarfile = os.path.join(tardir, "lava_results.tgz") - with image_partition_mounted(self._lava_image, self.root_part) as mntdir: + with image_partition_mounted(self._lava_image, self.config.root_part) as mntdir: logging_system( 'tar czf %s -C %s%s .' % ( tarfile, mntdir, self.context.lava_result_dir)) === modified file 'lava_dispatcher/config.py' --- lava_dispatcher/config.py 2012-09-11 21:00:08 +0000 +++ lava_dispatcher/config.py 2012-09-26 03:04:37 +0000 @@ -18,12 +18,95 @@ # along # with this program; if not, see . -from ConfigParser import ConfigParser, NoOptionError +from ConfigParser import ConfigParser import os import StringIO import logging +from configglue import parser, schema + +class DeviceSchema(schema.Schema): + android_binary_drivers = schema.StringOption() + boot_cmds = schema.StringOption(fatal=True) # Can do better here + boot_cmds_android = schema.StringOption(fatal=True) # And here + boot_cmds_oe = schema.StringOption(fatal=True) # And here? + boot_linaro_timeout = schema.IntOption(default=300) + boot_part = schema.IntOption(fatal=True) + boot_part_android_org = schema.StringOption() + bootloader_prompt = schema.StringOption() + cache_part_android_org = schema.StringOption() + client_type = schema.StringOption() + connection_command = schema.StringOption(fatal=True) + data_part_android = schema.StringOption() + data_part_android_org = schema.StringOption() + default_network_interface = schema.StringOption() + disablesuspend_timeout = schema.IntOption(default=240) + device_type = schema.StringOption(fatal=True) + enable_network_after_boot_android = schema.BoolOption(default=True) + git_url_disablesuspend_sh = schema.StringOption() + hard_reset_command = schema.StringOption() + hostname = schema.StringOption() + image_boot_msg = schema.StringOption() + interrupt_boot_command = schema.StringOption() + interrupt_boot_prompt = schema.StringOption() + lmc_dev_arg = schema.StringOption() + master_str = schema.StringOption() + pre_connect_command = schema.StringOption() + qemu_drive_interface = schema.StringOption() + qemu_machine_type = schema.StringOption() + reset_port_command = schema.StringOption() + root_part = schema.IntOption() + sdcard_part_android = schema.StringOption() + sdcard_part_android_org = schema.StringOption() + soft_boot_cmd = schema.StringOption() + sys_part_android = schema.StringOption() + sys_part_android_org = schema.StringOption() + tester_hostname = schema.StringOption(default="linaro") + tester_str = schema.StringOption() + val = schema.StringOption() + + simulator_binary = schema.StringOption() + license_server = schema.StringOption() + +class OptionDescriptor(object): + def __init__(self, name): + self.name = name + def __get__(self, inst, cls=None): + return inst.cp.get('__main__', self.name) + + +class DeviceConfig(object): + + def __init__(self, cp): + self.cp = cp + + for option in DeviceSchema().options(): + locals()[option.name] = OptionDescriptor(option.name) + + +class DispatcherSchema(schema.Schema): + default_qemu_binary = schema.StringOption(default="qemu") + lava_cachedir = schema.StringOption() + lava_image_tmpdir = schema.StringOption() + lava_image_url = schema.StringOption() + lava_proxy = schema.StringOption() + lava_result_dir = schema.StringOption() + lava_server_ip = schema.StringOption(fatal=True) + lava_test_deb = schema.StringOption() + lava_test_url = schema.StringOption() + logging_level = schema.IntOption() + + +class DispatcherConfig(object): + + def __init__(self, cp): + self.cp = cp + + for option in DispatcherSchema().options(): + locals()[option.name] = OptionDescriptor(option.name) + + default_config_path = os.path.join( os.path.dirname(__file__), 'default-config') @@ -44,13 +127,13 @@ def _read_into(path, cp): s = StringIO.StringIO() - s.write('[DEFAULT]\n') + s.write('[__main__]\n') s.write(open(path).read()) s.seek(0) cp.readfp(s) -def _get_config(name, config_dir, cp=None): +def _get_config(name, config_dir, cp): """Read a config file named name + '.conf'. This checks and loads files from the source tree, site wide location and @@ -65,55 +148,38 @@ if not config_files: raise Exception("no config files named %r found" % (name + ".conf")) config_files.reverse() - if cp is None: - cp = ConfigParser() logging.debug("About to read %s" % str(config_files)) for path in config_files: _read_into(path, cp) return cp -_sentinel = object() - -class ConfigWrapper(object): - def __init__(self, cp, config_dir): - self.cp = cp - self.config_dir = config_dir - def get(self, key, default=_sentinel): - try: - val = self.cp.get("DEFAULT", key) - if default is not _sentinel and val == '': - val = default - return val - except NoOptionError: - if default is not _sentinel: - return default - else: - raise - def getint(self, key, default=_sentinel): - try: - return self.cp.getint("DEFAULT", key) - except NoOptionError: - if default is not _sentinel: - return default - else: - raise - - def getboolean(self, key, default=True): - try: - return self.cp.getboolean("DEFAULT", key) - except ConfigParser.NoOptionError: - return default - -def get_config(name, config_dir): - return ConfigWrapper(_get_config(name, config_dir), config_dir) +def get_config(config_dir): + cp = parser.SchemaConfigParser(DispatcherSchema()) + _get_config("lava-dispatcher", config_dir, cp) + valid, report = cp.is_valid(report=True) + if not valid: + logging.warning("dispatcher config is not valid:\n %s", '\n '.join(report)) + c = DispatcherConfig(cp) + c.config_dir = config_dir + return c def get_device_config(name, config_dir): - device_config = _get_config("devices/%s" % name, config_dir) - cp = _get_config("device-defaults", config_dir) + # We read the device config once to get the device type, then we start + # again and read device-defaults, device-types/$device-type and + # devices/$device in that order. + initial_config = ConfigParser() + _get_config("devices/%s" % name, config_dir, initial_config) + + real_device_config = parser.SchemaConfigParser(DeviceSchema()) + _get_config("device-defaults", config_dir, real_device_config) _get_config( - "device-types/%s" % device_config.get('DEFAULT', 'device_type'), - config_dir, cp=cp) - _get_config("devices/%s" % name, config_dir, cp=cp) - cp.set("DEFAULT", "hostname", name) - return ConfigWrapper(cp, config_dir) + "device-types/%s" % initial_config.get('__main__', 'device_type'), + config_dir, real_device_config) + _get_config("devices/%s" % name, config_dir, real_device_config) + real_device_config.set("__main__", "hostname", name) + valid, report = real_device_config.is_valid(report=True) + if not valid: + logging.warning("Device config for %s is not valid:\n %s", name, '\n '.join(report)) + + return DeviceConfig(real_device_config) === modified file 'lava_dispatcher/context.py' --- lava_dispatcher/context.py 2012-07-30 17:03:46 +0000 +++ lava_dispatcher/context.py 2012-09-26 01:44:57 +0000 @@ -34,7 +34,7 @@ self.job_data = job_data device_config = get_device_config( target, dispatcher_config.config_dir) - client_type = device_config.get('client_type') + client_type = device_config.client_type if client_type == 'master' or client_type == 'conmux': self._client = LavaMasterImageClient(self, device_config) elif client_type == 'qemu': @@ -44,7 +44,7 @@ else: raise RuntimeError( "this version of lava-dispatcher only supports master, qemu, " - "and fastmodel clients, not %r" % device_config.get('client_type')) + "and fastmodel clients, not %r" % client_type) self.test_data = LavaTestData() self.oob_file = oob_file self._host_result_dir = None @@ -55,26 +55,6 @@ return self._client @property - def lava_server_ip(self): - return self.config.get("LAVA_SERVER_IP") - - @property - def lava_proxy(self): - return self.config.get("LAVA_PROXY", None) - - @property - def lava_cookies(self): - return self.config.get("LAVA_COOKIES", None) - - @property - def lava_image_tmpdir(self): - return self.config.get("LAVA_IMAGE_TMPDIR") - - @property - def lava_image_url(self): - return self.config.get("LAVA_IMAGE_URL") - - @property def any_host_bundles(self): return (self._host_result_dir is not None and len(os.listdir(self._host_result_dir)) > 0) @@ -84,11 +64,3 @@ if self._host_result_dir is None: self._host_result_dir = tempfile.mkdtemp() return self._host_result_dir - - @property - def lava_result_dir(self): - return self.config.get("LAVA_RESULT_DIR") - - @property - def lava_cachedir(self): - return self.config.get("LAVA_CACHEDIR") === modified file 'lava_dispatcher/default-config/lava-dispatcher/device-defaults.conf' --- lava_dispatcher/default-config/lava-dispatcher/device-defaults.conf 2012-08-08 22:31:12 +0000 +++ lava_dispatcher/default-config/lava-dispatcher/device-defaults.conf 2012-09-25 02:23:54 +0000 @@ -29,10 +29,10 @@ # XXX should be called # boot_android_test_image_commands ? boot_cmds_android = -# The device type. Settings in device-types/${TYPE}.conf override -# settings in this file, but are overridden by the +# The device type. Settings in device-types/${DEVICE_TYPE}.conf +# override settings in this file, but are overridden by the # devices/${DEVICE}.conf file. -type = +device_type = # The network interface that comes up by default default_network_interface = eth0 === modified file 'lava_dispatcher/tests/test_config.py' --- lava_dispatcher/tests/test_config.py 2012-09-12 18:09:46 +0000 +++ lava_dispatcher/tests/test_config.py 2012-09-26 02:55:25 +0000 @@ -57,17 +57,17 @@ "nocompcache vram=12M omapfb.debug=y " "omapfb.mode=dvi:1280x720MR-16@60'", "boot"] - uboot_cmds = beagle01_config.get("boot_cmds") + uboot_cmds = beagle01_config.boot_cmds self.assertEquals(expected, string_to_list(uboot_cmds)) def test_server_ip(self): - server_config = get_config("lava-dispatcher", test_config_dir) + server_config = get_config(test_config_dir) expected = "192.168.200.200" - lava_server_ip = server_config.get("LAVA_SERVER_IP") + lava_server_ip = server_config.lava_server_ip self.assertEqual(expected, lava_server_ip) def test_default_value_for_tester_hostname(self): create_config('devices/qemu01.conf', { 'device_type': 'qemu' }) config = get_device_config("qemu01", tmp_config_dir) client = LavaClient(None, config) - self.assertEqual('linaro', client.tester_hostname) + self.assertEqual('linaro', client.config.tester_hostname) === modified file 'setup.py' --- setup.py 2012-06-01 01:49:28 +0000 +++ setup.py 2012-09-26 02:12:19 +0000 @@ -32,6 +32,7 @@ "lava-tool >= 0.4", "lava-utils-interface", "pexpect >= 2.3", + "configglue", ], setup_requires=[ 'versiontools >= 1.8',