From patchwork Fri Aug 24 14:22:13 2012 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andy Doan X-Patchwork-Id: 10946 Return-Path: X-Original-To: patchwork@peony.canonical.com Delivered-To: patchwork@peony.canonical.com Received: from fiordland.canonical.com (fiordland.canonical.com [91.189.94.145]) by peony.canonical.com (Postfix) with ESMTP id F3C6C23E53 for ; Fri, 24 Aug 2012 14:22:17 +0000 (UTC) Received: from mail-iy0-f180.google.com (mail-iy0-f180.google.com [209.85.210.180]) by fiordland.canonical.com (Postfix) with ESMTP id D3317A18954 for ; Fri, 24 Aug 2012 14:21:59 +0000 (UTC) Received: by iafj25 with SMTP id j25so1198981iaf.11 for ; Fri, 24 Aug 2012 07:22:16 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=x-forwarded-to:x-forwarded-for:delivered-to:received-spf :content-type:mime-version:x-launchpad-project:x-launchpad-branch :x-launchpad-message-rationale:x-launchpad-branch-revision-number :x-launchpad-notification-type:to:from:subject:message-id:date :reply-to:sender:errors-to:precedence:x-generated-by :x-launchpad-hash:x-gm-message-state; bh=/2SqDJ97yQBvxnEH0QlM9f+4/XSZ2euMpC0ap9Og8Vo=; b=LDOFriHIuCZn4qtAUEddBbXD5cYFWHCot02YuXBOVyKDEBndnNUJLomLxJP7ga39c0 Ku6tt63a3Uc6Ah2VcPAugYDdZCaKBE+6xZOPZf97kDRz/mT/5lssrkErLXsSBRhNEy9f qYbeAES9z8PNJvg1sAJJ2lLpQjWdsh4g2olhyn1S+9JdLIxochocfNm/rSbWeyDSkN6u WryMlEItj4OwFF6HEPvH4caF8eg6iEIk+NNgzbpM6XHe4xnGF/Ui7S483iPQChbS3EoW Kdw/Asy5iH24ioLaXBHxFgCml2qooF+kglbTAu9XI5AcD/E4LlN18QVfvaP5h8F5l5by iUlQ== Received: by 10.50.45.162 with SMTP id o2mr2414275igm.0.1345818136774; Fri, 24 Aug 2012 07:22:16 -0700 (PDT) X-Forwarded-To: linaro-patchwork@canonical.com X-Forwarded-For: patch@linaro.org linaro-patchwork@canonical.com Delivered-To: patches@linaro.org Received: by 10.50.184.232 with SMTP id ex8csp322533igc; Fri, 24 Aug 2012 07:22:15 -0700 (PDT) Received: by 10.180.105.6 with SMTP id gi6mr5992904wib.4.1345818134596; Fri, 24 Aug 2012 07:22:14 -0700 (PDT) Received: from indium.canonical.com (indium.canonical.com. [91.189.90.7]) by mx.google.com with ESMTPS id r9si6070849wic.46.2012.08.24.07.22.13 (version=TLSv1/SSLv3 cipher=OTHER); Fri, 24 Aug 2012 07:22:14 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of bounces@canonical.com designates 91.189.90.7 as permitted sender) client-ip=91.189.90.7; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of bounces@canonical.com designates 91.189.90.7 as permitted sender) smtp.mail=bounces@canonical.com Received: from ackee.canonical.com ([91.189.89.26]) by indium.canonical.com with esmtp (Exim 4.71 #1 (Debian)) id 1T4um5-0008DB-Hq for ; Fri, 24 Aug 2012 14:22:13 +0000 Received: from ackee.canonical.com (localhost [127.0.0.1]) by ackee.canonical.com (Postfix) with ESMTP id 7BA32E01FC for ; Fri, 24 Aug 2012 14:22:13 +0000 (UTC) MIME-Version: 1.0 X-Launchpad-Project: lava-dispatcher X-Launchpad-Branch: ~linaro-validation/lava-dispatcher/trunk X-Launchpad-Message-Rationale: Subscriber X-Launchpad-Branch-Revision-Number: 374 X-Launchpad-Notification-Type: branch-revision To: Linaro Patch Tracker From: noreply@launchpad.net Subject: [Branch ~linaro-validation/lava-dispatcher/trunk] Rev 374: add a URL mapping feature Message-Id: <20120824142213.15181.55721.launchpad@ackee.canonical.com> Date: Fri, 24 Aug 2012 14:22:13 -0000 Reply-To: noreply@launchpad.net Sender: bounces@canonical.com Errors-To: bounces@canonical.com Precedence: bulk X-Generated-By: Launchpad (canonical.com); Revision="15843"; Instance="launchpad-lazr.conf" X-Launchpad-Hash: 2cb4fbff7a7cddb98f7f91030c050616f6b75a9f X-Gm-Message-State: ALoCoQnVjCT5MaHN5bJF9gBNxGJP95erG2CPYKIGUqW5dT0Hqivvw5zhGMtn287LHnj0JXaz/Jgb ------------------------------------------------------------ revno: 374 committer: Andy Doan branch nick: lava-dispatcher timestamp: Fri 2012-08-24 09:20:35 -0500 message: add a URL mapping feature This allows the dispatcher's downloading code to be able to override a URL given to it with another. For example, you might get a job that wants to download a file: http://blah/root.img This could be overriden to be file:///root.img You can specify one or more mappings in: /etc/lava-dispatcher/urlmappings.txt An example: ^https:\/\/.*\/precise\/restricted, scp://linaro-lava@mombin.canonical.com/precise/restricted/ ^https:\/\/.*\/~linaro-android-restricted\/, scp://linaro-lava@mombin.canonical.com/android/~linaro-android-restricted/ these take URLs that are protected by OpenID (not supported in the downloader) and replaces them with an SCP based URL modified: lava_dispatcher/downloader.py --- lp:lava-dispatcher https://code.launchpad.net/~linaro-validation/lava-dispatcher/trunk You are subscribed to branch lp:lava-dispatcher. To unsubscribe from this branch go to https://code.launchpad.net/~linaro-validation/lava-dispatcher/trunk/+edit-subscription === modified file 'lava_dispatcher/downloader.py' --- lava_dispatcher/downloader.py 2012-06-25 20:45:16 +0000 +++ lava_dispatcher/downloader.py 2012-08-24 14:20:35 +0000 @@ -23,6 +23,7 @@ import contextlib import logging import os +import re import shutil import subprocess import urllib2 @@ -107,6 +108,24 @@ filename = os.path.join(path, '.'.join(parts[:-1])) return (filename, suffix) +def _url_mapping(url, context): + '''allows the downloader to override a URL so that something like: + http://blah/ becomes file://localhost/blah + ''' + mappings = '%s/urlmappings.txt' % context.config.config_dir + if os.path.exists(mappings): + newurl = url + with open(mappings, 'r') as f: + for line in f.readlines(): + pat,rep = line.split(',') + pat = pat.strip() + rep = rep.strip() + newurl = re.sub(pat, rep, newurl) + if newurl != url: + url = newurl + logging.info('url mapped to: %s', url) + return url + def download_image(url, context, imgdir=None, delete_on_exit=True, decompress=True): '''downloads a image that's been compressed as .bz2 or .gz and @@ -118,6 +137,8 @@ if delete_on_exit: atexit.register(shutil.rmtree, imgdir) + url = _url_mapping(url, context) + url = urlparse.urlparse(url) stream = None if url.scheme == 'scp':