From patchwork Wed May 9 07:18:09 2012 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Le Chi Thu X-Patchwork-Id: 8483 Return-Path: X-Original-To: patchwork@peony.canonical.com Delivered-To: patchwork@peony.canonical.com Received: from fiordland.canonical.com (fiordland.canonical.com [91.189.94.145]) by peony.canonical.com (Postfix) with ESMTP id 9A69923F00 for ; Wed, 9 May 2012 07:18:13 +0000 (UTC) Received: from mail-gg0-f180.google.com (mail-gg0-f180.google.com [209.85.161.180]) by fiordland.canonical.com (Postfix) with ESMTP id 49E9BA180A9 for ; Wed, 9 May 2012 07:18:13 +0000 (UTC) Received: by gglu1 with SMTP id u1so2567573ggl.11 for ; Wed, 09 May 2012 00:18:12 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=x-forwarded-to:x-forwarded-for:delivered-to:received-spf :content-type:mime-version:x-launchpad-project:x-launchpad-branch :x-launchpad-message-rationale:x-launchpad-branch-revision-number :x-launchpad-notification-type:to:from:subject:message-id:date :reply-to:sender:errors-to:precedence:x-generated-by :x-launchpad-hash:x-gm-message-state; bh=N8WsDmAHBbnc2ouK4CLNn20v6E9Ckwo6MTMHZsqibN4=; b=K2WCuWFHa0pEfSWicg0BZvjmbWRtBNyUBxdreDwt6/Wl6AeLJGognw9XMESUlbNXnK dr1hnmUHl8Y3NJLuh/Zhks1kOERFReNWnLTYCpd9YdOy3IW5kTRGf4AKq/M1nJ+xhwvr DvhvSOs2Q/dU3QDicBNTJWyogFrk9AVJijnHhkuWhvgLFnEt+bQxevOhLPoA9tVROQjR gM74C8UqSdTQ6iMyLiPHbzx/w/qkgIGu+bcMDGtR2b7xENxwaf2yyGc3qvnrbmZbdrlp umAxpF7kN8HrIS37IBwPIoZNShRVboAmkbIYmiIbo5zwBhkAMXnFql5huLAdBa16KuDU /mkQ== Received: by 10.50.41.196 with SMTP id h4mr2736868igl.33.1336547892683; Wed, 09 May 2012 00:18:12 -0700 (PDT) X-Forwarded-To: linaro-patchwork@canonical.com X-Forwarded-For: patch@linaro.org linaro-patchwork@canonical.com Delivered-To: patches@linaro.org Received: by 10.231.73.147 with SMTP id q19csp253848ibj; Wed, 9 May 2012 00:18:11 -0700 (PDT) Received: by 10.180.75.241 with SMTP id f17mr51610246wiw.11.1336547890436; Wed, 09 May 2012 00:18:10 -0700 (PDT) Received: from indium.canonical.com (indium.canonical.com. [91.189.90.7]) by mx.google.com with ESMTPS id gg1si17284831wib.46.2012.05.09.00.18.09 (version=TLSv1/SSLv3 cipher=OTHER); Wed, 09 May 2012 00:18:10 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of bounces@canonical.com designates 91.189.90.7 as permitted sender) client-ip=91.189.90.7; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of bounces@canonical.com designates 91.189.90.7 as permitted sender) smtp.mail=bounces@canonical.com Received: from ackee.canonical.com ([91.189.89.26]) by indium.canonical.com with esmtp (Exim 4.71 #1 (Debian)) id 1SS1A1-0001kb-Qt for ; Wed, 09 May 2012 07:18:09 +0000 Received: from ackee.canonical.com (localhost [127.0.0.1]) by ackee.canonical.com (Postfix) with ESMTP id C02B9E1F60 for ; Wed, 9 May 2012 07:18:09 +0000 (UTC) MIME-Version: 1.0 X-Launchpad-Project: lava-dispatcher X-Launchpad-Branch: ~linaro-validation/lava-dispatcher/trunk X-Launchpad-Message-Rationale: Subscriber X-Launchpad-Branch-Revision-Number: 285 X-Launchpad-Notification-Type: branch-revision To: Linaro Patch Tracker From: noreply@launchpad.net Subject: [Branch ~linaro-validation/lava-dispatcher/trunk] Rev 285: Merge revert-cache-tarball Message-Id: <20120509071809.18536.44924.launchpad@ackee.canonical.com> Date: Wed, 09 May 2012 07:18:09 -0000 Reply-To: noreply@launchpad.net Sender: bounces@canonical.com Errors-To: bounces@canonical.com Precedence: bulk X-Generated-By: Launchpad (canonical.com); Revision="15213"; Instance="launchpad-lazr.conf" X-Launchpad-Hash: 79cc9e4e1cc6c40454f575515b72a762425d74b5 X-Gm-Message-State: ALoCoQk2si194WU/wdd0EI7YYu8tHWn/G6LGoZODOapf+LdW2lIKRTcfo+Ujlj9BJCP7uOkwn/RY Merge authors: Spring Zhang (qzhang) Related merge proposals: https://code.launchpad.net/~qzhang/lava-dispatcher/revert-cache-tarballs/+merge/105029 proposed by: Spring Zhang (qzhang) review: Approve - Michael Hudson-Doyle (mwhudson) review: Approve - Le Chi Thu (le-chi-thu) ------------------------------------------------------------ revno: 285 [merge] committer: Le Chi Thu le.chi.thu@linaro.org branch nick: trunk timestamp: Wed 2012-05-09 09:16:32 +0200 message: Merge revert-cache-tarball modified: doc/changes.rst lava_dispatcher/client/master.py lava_dispatcher/context.py lava_dispatcher/default-config/lava-dispatcher/lava-dispatcher.conf lava_dispatcher/utils.py --- lp:lava-dispatcher https://code.launchpad.net/~linaro-validation/lava-dispatcher/trunk You are subscribed to branch lp:lava-dispatcher. To unsubscribe from this branch go to https://code.launchpad.net/~linaro-validation/lava-dispatcher/trunk/+edit-subscription === modified file 'doc/changes.rst' --- doc/changes.rst 2012-05-06 16:01:25 +0000 +++ doc/changes.rst 2012-05-09 07:16:32 +0000 @@ -7,6 +7,7 @@ ========================================= * Use squid proxy for caching mechanism +* Merge revert-cache-tarball .. _version_0_6: === modified file 'lava_dispatcher/client/master.py' --- lava_dispatcher/client/master.py 2012-05-08 18:35:36 +0000 +++ lava_dispatcher/client/master.py 2012-05-09 07:16:32 +0000 @@ -35,9 +35,11 @@ from lava_dispatcher.utils import ( download, + download_with_cache, logging_spawn, logging_system, - string_to_list) + string_to_list, + url_to_cache, link_or_copy_file) from lava_dispatcher.client.base import ( CommandRunner, CriticalError, @@ -293,12 +295,82 @@ return uncompressed_name return image_file + def _tarball_url_to_cache(self, url, cachedir): + cache_loc = url_to_cache(url, cachedir) + # can't have a folder name same as file name. replacing '.' with '.' + return os.path.join(cache_loc.replace('.','-'), "tarballs") + + def _are_tarballs_cached(self, image, lava_cachedir): + cache_loc = self._tarball_url_to_cache(image, lava_cachedir) + cached = os.path.exists(os.path.join(cache_loc, "boot.tgz")) and \ + os.path.exists(os.path.join(cache_loc, "root.tgz")) + + if cached: + return True; + + # Check if there is an other lava-dispatch instance have start to cache the same image + # see the _about_to_cache_tarballs + if not os.path.exists(os.path.join(cache_loc, "tarballs-cache-ongoing")): + return False + + # wait x minute for caching is done. + waittime=20 + + logging.info("Waiting for the other instance of lava-dispatcher to finish the caching of %s", image) + while waittime > 0: + if not os.path.exists(os.path.join(cache_loc, "tarballs-cache-ongoing")): + waittime = 0 + else: + time.sleep(60) + waittime = waittime - 1 + if (waittime % 5) == 0: + logging.info("%d minute left..." % waittime) + + return os.path.exists(os.path.join(cache_loc, "boot.tgz")) and \ + os.path.exists(os.path.join(cache_loc, "root.tgz")) + + def _get_cached_tarballs(self, image, tarball_dir, lava_cachedir): + cache_loc = self._tarball_url_to_cache(image, lava_cachedir) + + boot_tgz = os.path.join(tarball_dir,"boot.tgz") + root_tgz = os.path.join(tarball_dir,"root.tgz") + link_or_copy_file(os.path.join(cache_loc, "root.tgz"), root_tgz) + link_or_copy_file(os.path.join(cache_loc, "boot.tgz"), boot_tgz) + + return (boot_tgz,root_tgz) + + def _about_to_cache_tarballs(self, image, lava_cachedir): + # create this folder to indicate this instance of lava-dispatcher is caching this image. + # see _are_tarballs_cached + # return false if unable to create the directory. The caller should not cache the tarballs + cache_loc = self._tarball_url_to_cache(image, lava_cachedir) + path = os.path.join(cache_loc, "tarballs-cache-ongoing") + try: + os.makedirs(path) + except OSError as exc: # Python >2.5 + if exc.errno == errno.EEXIST: + # other dispatcher process already caching - concurrency issue + return False + else: + raise + return True + + def _cache_tarballs(self, image, boot_tgz, root_tgz, lava_cachedir): + cache_loc = self._tarball_url_to_cache(image, lava_cachedir) + if not os.path.exists(cache_loc): + os.makedirs(cache_loc) + c_boot_tgz = os.path.join(cache_loc, "boot.tgz") + c_root_tgz = os.path.join(cache_loc, "root.tgz") + shutil.copy(boot_tgz, c_boot_tgz) + shutil.copy(root_tgz, c_root_tgz) + path = os.path.join(cache_loc, "tarballs-cache-ongoing") + if os.path.exists(path): + shutil.rmtree(path) + def deploy_linaro(self, hwpack=None, rootfs=None, image=None, kernel_matrix=None, use_cache=True, rootfstype='ext3'): LAVA_IMAGE_TMPDIR = self.context.lava_image_tmpdir LAVA_IMAGE_URL = self.context.lava_image_url - lava_proxy = self.context.lava_proxy - # validate in parameters if image is None: if hwpack is None or rootfs is None: @@ -317,10 +389,35 @@ else: tarball_dir = mkdtemp(dir=LAVA_IMAGE_TMPDIR) os.chmod(tarball_dir, 0755) - proxy = lava_proxy if use_cache else None - image_file = download(image, tarball_dir, proxy) - image_file = self.decompress(image_file) - boot_tgz, root_tgz = self._generate_tarballs(image_file) + if use_cache: + lava_cachedir = self.context.lava_cachedir + if self._are_tarballs_cached(image, lava_cachedir): + logging.info("Reusing cached tarballs") + boot_tgz, root_tgz = self._get_cached_tarballs(image, tarball_dir, lava_cachedir) + else: + logging.info("Downloading and caching the tarballs") + # in some corner case, there can be more than one lava-dispatchers execute + # caching of same tarballs exact at the same time. One of them will successfully + # get the lock directory. The rest will skip the caching if _about_to_cache_tarballs + # return false. + should_cache = self._about_to_cache_tarballs(image, lava_cachedir) + image_file = download_with_cache(image, tarball_dir, lava_cachedir) + image_file = self.decompress(image_file) + boot_tgz, root_tgz = self._generate_tarballs(image_file) + if should_cache: + self._cache_tarballs(image, boot_tgz, root_tgz, lava_cachedir) + else: + lava_proxy = self.context.lava_proxy + image_file = download(image, tarball_dir, lava_proxy) + image_file = self.decompress(image_file) + boot_tgz, root_tgz = self._generate_tarballs(image_file) + # remove the cached tarballs + cache_loc = self._tarball_url_to_cache(image, lava_cachedir) + shutil.rmtree(cache_loc, ignore_errors = true) + # remove the cached image files + cache_loc = url_to_cache + shutil.rmtree(cache_loc, ignore_errors = true) + except CriticalError: raise except: @@ -441,6 +538,7 @@ logging.info("Downloading the image files") proxy = lava_proxy if use_cache else None + boot_path = download(boot_url, tarball_dir, proxy) system_path = download(system_url, tarball_dir, proxy) data_path = download(data_url, tarball_dir, proxy) === modified file 'lava_dispatcher/context.py' --- lava_dispatcher/context.py 2012-05-07 03:46:16 +0000 +++ lava_dispatcher/context.py 2012-05-08 05:37:59 +0000 @@ -84,3 +84,7 @@ @property def lava_result_dir(self): return self.config.get("LAVA_RESULT_DIR") + + @property + def lava_cachedir(self): + return self.config.get("LAVA_CACHEDIR") === modified file 'lava_dispatcher/default-config/lava-dispatcher/lava-dispatcher.conf' --- lava_dispatcher/default-config/lava-dispatcher/lava-dispatcher.conf 2012-05-07 03:46:16 +0000 +++ lava_dispatcher/default-config/lava-dispatcher/lava-dispatcher.conf 2012-05-08 05:37:59 +0000 @@ -18,6 +18,9 @@ # Location on the device for storing test results. LAVA_RESULT_DIR = /lava/results +# Location for caching downloaded artifacts such as hwpacks and images +LAVA_CACHEDIR = /linaro/images/cache + # Python logging level to use # 10 = DEBUG # 20 = INFO === modified file 'lava_dispatcher/utils.py' --- lava_dispatcher/utils.py 2012-05-06 15:58:03 +0000 +++ lava_dispatcher/utils.py 2012-05-08 05:56:59 +0000 @@ -53,9 +53,46 @@ raise RuntimeError("Could not retrieve %s" % url) return filename +def link_or_copy_file(src, dest): + try: + dir = os.path.dirname(dest) + if not os.path.exists(dir): + os.makedirs(dir) + os.link(src, dest) + except OSError, err: + if err.errno == errno.EXDEV: + shutil.copy(src, dest) + if err.errno == errno.EEXIST: + logging.debug("Cached copy of %s already exists" % dest) + else: + logging.exception("os.link '%s' with '%s' failed" % (src, dest)) + +def copy_file(src, dest): + dir = os.path.dirname(dest) + if not os.path.exists(dir): + os.makedirs(dir) + shutil.copy(src, dest) + + # XXX: duplication, we have similar code in lava-test, we need to move that to # lava.utils -> namespace as standalone package -# def download_with_cache(url, path="", cachedir=""): +def download_with_cache(url, path="", cachedir=""): + cache_loc = url_to_cache(url, cachedir) + if os.path.exists(cache_loc): + filename = os.path.basename(cache_loc) + file_location = os.path.join(path, filename) + link_or_copy_file(cache_loc, file_location) + else: + file_location = download(url, path) + copy_file(file_location, cache_loc) + return file_location + + +def url_to_cache(url, cachedir): + url_parts = urlparse.urlsplit(url) + path = os.path.join(cachedir, url_parts.netloc, + url_parts.path.lstrip(os.sep)) + return path def string_to_list(string):