From patchwork Wed Feb 19 16:35:30 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Robert Foley X-Patchwork-Id: 183634 Delivered-To: patch@linaro.org Received: by 2002:a92:1f12:0:0:0:0:0 with SMTP id i18csp669910ile; Wed, 19 Feb 2020 08:50:58 -0800 (PST) X-Google-Smtp-Source: APXvYqz1RuwKBljVF8azJAIdTKKDpAC989o/fJ4jyRgYyqzJyUcRAdYe5S5poXjCLKdCWako9OEr X-Received: by 2002:a7b:c14d:: with SMTP id z13mr11295404wmi.71.1582131058038; Wed, 19 Feb 2020 08:50:58 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1582131058; cv=none; d=google.com; s=arc-20160816; b=DqBvHfyzye+zKKxSygc+4CML7DN5SBr3t1IjGPvgoL6kTe9uuWf+/7eQAGzsUNwgRT p+NcyoFCoWwX5imj+lNVYnEA1S9RvawY3CgCsX7H8HW0B8EaTuau/6SHJohNg8UjFfil 2wDaRbWtyI5i1QeUOquuw+wi73V4pFtdgVFlX3aEW2V5A5Qm3oqilmjEXaQVY5sU4+80 njCc9P8gQfhA4r40SUJ/HPIRNbrVEN9Bg49eK1Tw5xZ2RkNHKNKw7w98MsvoxMzYJ9JT fPl47O4BKUxN4UsU76q9S7k5Z5DmWDKEBxF9uKAJrWdPP/43eU18HFXv6yN9GGPpqhrA dk+A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:cc:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:references:in-reply-to :message-id:date:subject:to:from:dkim-signature; bh=36Q8JSf0N/o+sezmkSBG9jdKzoyNhca6OBIv/UTtZJ4=; b=rPzWX2dqAX+twu1QmxBO5wjvk5i+Q3bxcwrSD2D7pMJ82U1iAM3uCVXvrKJ4oh5ywM 64BmWFtr/x4rRsxtDpUuOrAFEAPhnEbxeiQZDV320lwvr5wDtGoqXw/J4rbLUYSFH1Jk dSna/Z6OQk1IW77YKOJoRDMB1EoEbg/2A7qEuPrgdPRRhfnla/1kJ4I3nuXb+xR1gSrz VAC7S5DeufG+vFllle02dOlnp4B/qPUcBM7aMgFhNYVL5nkQj08XUPT9TP2TqsIAloYw cnyrYzgvZSVva8dPRMTlRhZiF7MHm0A4L/AMXw/tEiNtaWNJeMxi0oxMYtY7BAtV54AY r3aA== ARC-Authentication-Results: i=1; mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=AA8aXCXt; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id j7si214006wrp.326.2020.02.19.08.50.57 for (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 19 Feb 2020 08:50:58 -0800 (PST) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=AA8aXCXt; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:56360 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1j4SYq-00028n-Pz for patch@linaro.org; Wed, 19 Feb 2020 11:50:56 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:41972) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1j4SOH-00037m-DC for qemu-devel@nongnu.org; Wed, 19 Feb 2020 11:40:03 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1j4SOF-00083t-MV for qemu-devel@nongnu.org; Wed, 19 Feb 2020 11:40:01 -0500 Received: from mail-pf1-x42b.google.com ([2607:f8b0:4864:20::42b]:33699) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1j4SOF-00083K-Dl for qemu-devel@nongnu.org; Wed, 19 Feb 2020 11:39:59 -0500 Received: by mail-pf1-x42b.google.com with SMTP id n7so341497pfn.0 for ; Wed, 19 Feb 2020 08:39:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=36Q8JSf0N/o+sezmkSBG9jdKzoyNhca6OBIv/UTtZJ4=; b=AA8aXCXtJGNvVvQcMNARMLwDKoF7WNKiyp5laffKcMfqz7AlPPsLg4Zn0I7LMB0xOj QAnDAqttd/JdhrFrd4RWUcdgqxRCBZFEZ+fUOgOc4zq35Tfp/zkC1oiJ+7sLwjReyk1K alzBQ3UjvhySvbTWvhXezRGwprqxOgWJZdFDe4qL1g3Zpv4ntWRcfpa9DxmBGFbF+WgA 2NMUK5+uuKuG2hb7qI+SvBON9LvZikvJ445tDJBC2v4EuJRyJppxDdczCztgzn7jJTW/ JSsT0emWW3CP6EvjB4gCXHNhIe0NrsMeompRItWCu3KU3FqShsmqdwicH+GNadGcT/Ry Hqvw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=36Q8JSf0N/o+sezmkSBG9jdKzoyNhca6OBIv/UTtZJ4=; b=aXnf8ekBJa9LDHuy/mevXzwBEYoyoe5mhnJ5D6PclyMBdyQyiw2y1k4O/Igypj9tFa fpA0/fd9K/6qTwa+f9zdj7YkiWSAZKWNAbByPKJBKCuQN7DTpWjbk8hHrptG/Ip5H7Eg fGH2I/wjpnzgGj3Ftywf/HhnTjfiKKEXLXZbElJ5DnjYREnWN5Ss6ki9GYDcn/eK085P nkcZIWfMujROZPoX0JZ1/weyDRe0H1KOyvk84ooUGJiG90DWmmXayTH0UYh5WJrEpfFg SBnfB+tRdm9R40IRk/i93Dj7Kff0c2BWsIBmDkvr/EAQg/Cjqe5HLro9pWZk7CnpZh6s D7WA== X-Gm-Message-State: APjAAAVNpvxuZ6o0YB+Ec6o8rexrqxwXRsoyQnFavTBtPBx1lM1Xzy0d VB5YbzR7i+a8rvqpm6zGKqn1dSgMozg= X-Received: by 2002:a63:d94d:: with SMTP id e13mr28064012pgj.240.1582130397876; Wed, 19 Feb 2020 08:39:57 -0800 (PST) Received: from Rfoley-MA01.hsd1.ma.comcast.net ([2601:199:4480:60c0:3cc2:5eb7:e248:ad23]) by smtp.gmail.com with ESMTPSA id a69sm112362pfa.129.2020.02.19.08.39.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 19 Feb 2020 08:39:57 -0800 (PST) From: Robert Foley To: qemu-devel@nongnu.org Subject: [PATCH v2 07/14] tests/vm: Add configuration to basevm.py Date: Wed, 19 Feb 2020 11:35:30 -0500 Message-Id: <20200219163537.22098-8-robert.foley@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200219163537.22098-1-robert.foley@linaro.org> References: <20200219163537.22098-1-robert.foley@linaro.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::42b X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: philmd@redhat.com, alex.bennee@linaro.org, robert.foley@linaro.org, peter.puhov@linaro.org Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" Added use of a configuration to tests/vm/basevm.py. The configuration provides parameters used to configure a VM. This allows for providing alternate configurations to the VM being created/launched. cpu, machine, memory, and NUMA configuration are all examples of configuration which we might want to vary on the VM being created or launched. This will for example allow for creating an aarch64 vm. Signed-off-by: Robert Foley --- tests/vm/basevm.py | 143 +++++++++++++++++++++++++++++++++++---------- 1 file changed, 112 insertions(+), 31 deletions(-) -- 2.17.1 diff --git a/tests/vm/basevm.py b/tests/vm/basevm.py index a22b59b23c..ed1b56bcea 100644 --- a/tests/vm/basevm.py +++ b/tests/vm/basevm.py @@ -30,15 +30,39 @@ import shutil import multiprocessing import traceback -SSH_KEY = open(os.path.join(os.path.dirname(__file__), - "..", "keys", "id_rsa")).read() -SSH_PUB_KEY = open(os.path.join(os.path.dirname(__file__), - "..", "keys", "id_rsa.pub")).read() - +SSH_KEY_FILE = os.path.join(os.path.dirname(__file__), + "..", "keys", "id_rsa") +SSH_PUB_KEY_FILE = os.path.join(os.path.dirname(__file__), + "..", "keys", "id_rsa.pub") + +# This is the standard configuration. +# Any or all of these can be overridden by +# passing in a config argument to the VM constructor. +DEFAULT_CONFIG = { + 'cpu' : "max", + 'machine' : 'pc', + 'guest_user' : "qemu", + 'guest_pass' : "qemupass", + 'root_pass' : "qemupass", + 'ssh_key_file' : SSH_KEY_FILE, + 'ssh_pub_key_file': SSH_PUB_KEY_FILE, + 'memory' : "4G", + 'extra_args' : [], + 'qemu_args' : "", + 'dns' : "", + 'ssh_port' : 0, + 'install_cmds' : "", + 'boot_dev_type' : "block", + 'ssh_timeout' : 1, +} +BOOT_DEVICE = { + 'block' : "-drive file={},if=none,id=drive0,cache=writeback "\ + "-device virtio-blk,drive=drive0,bootindex=0", + 'scsi' : "-device virtio-scsi-device,id=scsi "\ + "-drive file={},format=raw,if=none,id=hd0 "\ + "-device scsi-hd,drive=hd0,bootindex=0", +} class BaseVM(object): - GUEST_USER = "qemu" - GUEST_PASS = "qemupass" - ROOT_PASS = "qemupass" envvars = [ "https_proxy", @@ -63,20 +87,31 @@ class BaseVM(object): # 4 is arbitrary, but greater than 2, # since we found we need to wait more than twice as long. tcg_ssh_timeout_multiplier = 4 - def __init__(self, debug=False, vcpus=None, + def __init__(self, debug=False, vcpus=None, config=None, log_console=False): self._guest = None + # Allow input config to override defaults. + self._config = DEFAULT_CONFIG.copy() + if config != None: + self._config.update(config) + self.validate_ssh_keys() self._tmpdir = os.path.realpath(tempfile.mkdtemp(prefix="vm-test-", suffix=".tmp", dir=".")) atexit.register(shutil.rmtree, self._tmpdir) - - self._ssh_key_file = os.path.join(self._tmpdir, "id_rsa") - open(self._ssh_key_file, "w").write(SSH_KEY) - subprocess.check_call(["chmod", "600", self._ssh_key_file]) - - self._ssh_pub_key_file = os.path.join(self._tmpdir, "id_rsa.pub") - open(self._ssh_pub_key_file, "w").write(SSH_PUB_KEY) + # Copy the key files to a temporary directory. + # Also chmod the key file to agree with ssh requirements. + self._config['ssh_key'] = \ + open(self._config['ssh_key_file']).read().rstrip() + self._config['ssh_pub_key'] = \ + open(self._config['ssh_pub_key_file']).read().rstrip() + self._ssh_tmp_key_file = os.path.join(self._tmpdir, "id_rsa") + open(self._ssh_tmp_key_file, "w").write(self._config['ssh_key']) + subprocess.check_call(["chmod", "600", self._ssh_tmp_key_file]) + + self._ssh_tmp_pub_key_file = os.path.join(self._tmpdir, "id_rsa.pub") + open(self._ssh_tmp_pub_key_file, + "w").write(self._config['ssh_pub_key']) self._console_log_path = None if log_console: @@ -90,11 +125,14 @@ class BaseVM(object): self._stdout = sys.stdout else: self._stdout = self._devnull + netdev = "user,id=vnet,hostfwd=:127.0.0.1:{}-:22" self._args = [ \ - "-nodefaults", "-m", "4G", - "-cpu", "max", - "-netdev", "user,id=vnet,hostfwd=:127.0.0.1:0-:22" + - (",ipv6=no" if not self.ipv6 else ""), + "-nodefaults", "-m", self._config['memory'], + "-cpu", self._config['cpu'], + "-netdev", + netdev.format(self._config['ssh_port']) + + (",ipv6=no" if not self.ipv6 else "") + + (",dns=" + self._config['dns'] if self._config['dns'] else ""), "-device", "virtio-net-pci,netdev=vnet", "-vnc", "127.0.0.1:0,to=20"] if vcpus and vcpus > 1: @@ -105,6 +143,45 @@ class BaseVM(object): logging.info("KVM not available, not using -enable-kvm") self._data_args = [] + if self._config['qemu_args'] != None: + qemu_args = self._config['qemu_args'] + qemu_args = qemu_args.replace('\n',' ').replace('\r','') + # Remove any empty strings from list. + self._config['extra_args'] = [x for x in qemu_args.split(' ') if x] + + def validate_ssh_keys(self): + """Check to see if the ssh key files exist.""" + if 'ssh_key_file' not in self._config or\ + not os.path.exists(self._config['ssh_key_file']): + raise Exception("ssh key file not found.") + if 'ssh_pub_key_file' not in self._config or\ + not os.path.exists(self._config['ssh_pub_key_file']): + raise Exception("ssh pub key file not found.") + + def wait_boot(self, wait_string=None): + """Wait for the standard string we expect + on completion of a normal boot. + The user can also choose to override with an + alternate string to wait for.""" + if wait_string is None: + if self.login_prompt is None: + raise Exception("self.login_prompt not defined") + wait_string = self.login_prompt + # Intentionally bump up the default timeout under TCG, + # since the console wait below takes longer. + timeout = self.socket_timeout + if not kvm_available(self.arch): + timeout *= 8 + self.console_init(timeout=timeout) + self.console_wait(wait_string) + + def __getattr__(self, name): + # Support direct access to config by key. + # for example, access self._config['cpu'] by self.cpu + if name.lower() in self._config.keys(): + return self._config[name.lower()] + return object.__getattribute__(self, name) + def _download_with_cache(self, url, sha256sum=None, sha512sum=None): def check_sha256sum(fname): if not sha256sum: @@ -136,8 +213,9 @@ class BaseVM(object): "-t", "-o", "StrictHostKeyChecking=no", "-o", "UserKnownHostsFile=" + os.devnull, - "-o", "ConnectTimeout=1", - "-p", self.ssh_port, "-i", self._ssh_key_file] + "-o", + "ConnectTimeout={}".format(self._config["ssh_timeout"]), + "-p", self.ssh_port, "-i", self._ssh_tmp_key_file] # If not in debug mode, set ssh to quiet mode to # avoid printing the results of commands. if not self.debug: @@ -186,16 +264,16 @@ class BaseVM(object): "virtio-blk,drive=%s,serial=%s,bootindex=1" % (name, name)] def boot(self, img, extra_args=[]): - args = self._args + [ - "-device", "VGA", - "-drive", "file=%s,if=none,id=drive0,cache=writeback" % img, - "-device", "virtio-blk,drive=drive0,bootindex=0"] - args += self._data_args + extra_args + boot_dev = BOOT_DEVICE[self._config['boot_dev_type']] + boot_params = boot_dev.format(img) + args = self._args + boot_params.split(' ') + args += self._data_args + extra_args + self._config['extra_args'] + args += ["-device", "VGA"] logging.debug("QEMU args: %s", " ".join(args)) qemu_bin = os.environ.get("QEMU", "qemu-system-" + self.arch) guest = QEMUMachine(binary=qemu_bin, args=args, console_log=self._console_log_path) - guest.set_machine('pc') + guest.set_machine(self._config['machine']) guest.set_console() try: guest.launch() @@ -307,7 +385,8 @@ class BaseVM(object): self.console_send(command) def console_ssh_init(self, prompt, user, pw): - sshkey_cmd = "echo '%s' > .ssh/authorized_keys\n" % SSH_PUB_KEY.rstrip() + sshkey_cmd = "echo '%s' > .ssh/authorized_keys\n" \ + % self._config['ssh_pub_key'].rstrip() self.console_wait_send("login:", "%s\n" % user) self.console_wait_send("Password:", "%s\n" % pw) self.console_wait_send(prompt, "mkdir .ssh\n") @@ -437,15 +516,17 @@ def parse_args(vmcls): parser.disable_interspersed_args() return parser.parse_args() -def main(vmcls): +def main(vmcls, config=None): try: + if config == None: + config = {} args, argv = parse_args(vmcls) if not argv and not args.build_qemu and not args.build_image: print("Nothing to do?") return 1 logging.basicConfig(level=(logging.DEBUG if args.debug else logging.WARN)) - vm = vmcls(debug=args.debug, vcpus=args.jobs, + vm = vmcls(debug=args.debug, vcpus=args.jobs, config=config, log_console=args.log_console) if args.build_image: if os.path.exists(args.image) and not args.force: