From patchwork Mon Jul 13 20:04:07 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Alex_Benn=C3=A9e?= X-Patchwork-Id: 235492 Delivered-To: patch@linaro.org Received: by 2002:a92:d244:0:0:0:0:0 with SMTP id v4csp3138719ilg; Mon, 13 Jul 2020 13:07:56 -0700 (PDT) X-Google-Smtp-Source: ABdhPJyvdaJmCuGtQZVuBO6GAnRj+NBEOIcXliQysd5+HMfXUP0E2MnH+ABmvG6OGu7MuEBAlbjQ X-Received: by 2002:a25:cc12:: with SMTP id l18mr2488546ybf.480.1594670876417; Mon, 13 Jul 2020 13:07:56 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1594670876; cv=none; d=google.com; s=arc-20160816; b=M+UpbktOBye9K4L2f5N3Fx3FlTLHVgrEOKNYDQeiMd4nhIM3R/jkqx+PG4LLdQwzOK hPzBajHIDLq5OpNAmKuijFQFOWvRfT4cdBVt7z0qB7Db0nkKtdxiw34Zb52U5BZs5eiD ZrjcEbEkATGB8nrr8YC6yU+xOODJsFCsUOotKNa5KXocs8gTDQVs+1u3OiquJDBU8lTB ryNzHD3Iupg8vxpEcyls//q9YMLnYVQqg/XEO0Az9ecjvtOC+pEle4Zr5w//KYPd7vrL iKFrQ4yllww7kCPeZ7CWNtbYOwxl0xnt5+047ot6Fyrv4+VQ3l32OAP7PPBDXHMPRe4L zT+A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:cc:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:to:from :dkim-signature; bh=zYmy6YbAdYCCn5X7N7JnznTD44q8kk8nDaLydetiUF4=; b=q/cnR6kT8xkW7VNNURsnGz/kViLXEn8rNqAY/0U9wVo79oV8Aqj1Hsyp9c79EbrANi i8OUuB06JoRZPIW1SjojQo+yApj+X+pd8mn9HxvK5I9RmwhiGBPRBuSX00dENs+yZpew OdEwvADhtV8ZdjaLh5XML0C8dN7Rl5zxGKvttN0+qJIbhik6BNCZQY/5Ok/XJH2/vJ20 7kn8AU/U8qXHC2zHWxRs8ASQdR7oHI8mkZUqv4qpEj9oWreZoMYVU+g9Yehlotb1cMEc 5Afl/c41y3GVrdOSuEyETs/PShHS9yj1ka6JGNo1hrQsm5TdU8PTp+if/h1S+XXyTWU8 BACQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=iRoIvctJ; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id i3si15923021ybq.145.2020.07.13.13.07.56 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Mon, 13 Jul 2020 13:07:56 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=iRoIvctJ; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:51836 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jv4jz-0005RB-Tv for patch@linaro.org; Mon, 13 Jul 2020 16:07:55 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:59448) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jv4gb-0007zx-Qe for qemu-devel@nongnu.org; Mon, 13 Jul 2020 16:04:25 -0400 Received: from mail-wm1-x344.google.com ([2a00:1450:4864:20::344]:50452) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1jv4ga-0006Gg-6W for qemu-devel@nongnu.org; Mon, 13 Jul 2020 16:04:25 -0400 Received: by mail-wm1-x344.google.com with SMTP id l17so1153429wmj.0 for ; Mon, 13 Jul 2020 13:04:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=zYmy6YbAdYCCn5X7N7JnznTD44q8kk8nDaLydetiUF4=; b=iRoIvctJRweIWyuMWsjAtE8+JehMQIOHYYgkXVVIUgeOajqBilO/1mv4VT3kLduKqH Rdg5OhIxUXu7LbK67T5qfGAPbc2BUbtDrNnS0g5Axfio4PVe3bsdDRes7jHLLZzXXm/z iIwMbORvDonXZlZqGuI3boDjSOFy/TfKyTFhpeEjqXOBFm3V4aoU4EvfHqGD2/4uCnNu i53jSVpmZRtkTVQEqACbZGjXNMoO0NUCDuV2sfHuMjOcGd6UK+WfBAudmxK6PM1CzSE5 RMC3OgP6rcORKhr3QL0sia9O8g+pg9ih25NnHtU4zlCEir5cR3mcs2+k+6c+OgHOqhE+ sDHA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=zYmy6YbAdYCCn5X7N7JnznTD44q8kk8nDaLydetiUF4=; b=HNqmLK5dHVdgpgJtIVE1zXsqWdj3j/UAtVUqro/3EibFBXt8iH1Gjv1i41JCGd0/cc 0JQvlTLt2W6FnXvpzW+2gPcC7FeRqL9W4cgTUo6bndBOzUo4xQr2DyK2FrM+Oaa6bs0i KAzgN+npg6dzn6DCU+A2i0HfblM4ZpZdpspGjkMRyM6xVRhDYgsUn+r3sDY7InccoOzH heCOPWA5aOKi7JYrCvPTlXt9/swTj9sL4O3a9SXzlHuM7miC3vVuOeFGs1rrzRE9WtII gFnV+0Ro8eOrwR562iczceVelI5foC3mcZqItTFTqz/rDIoSXFqvjxIrczNluOULO4rs akTA== X-Gm-Message-State: AOAM5334BpKciG5PG37g1NI5H919uEpnFBDadrtn0j3xmfijB8IWH+PM oBD8ZcOX21rQf0Q8b+/44/ZBQw== X-Received: by 2002:a1c:a557:: with SMTP id o84mr1071156wme.42.1594670662723; Mon, 13 Jul 2020 13:04:22 -0700 (PDT) Received: from zen.linaroharston ([51.148.130.216]) by smtp.gmail.com with ESMTPSA id w16sm30261370wrg.95.2020.07.13.13.04.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 13 Jul 2020 13:04:16 -0700 (PDT) Received: from zen.lan (localhost [127.0.0.1]) by zen.linaroharston (Postfix) with ESMTP id D86141FF8F; Mon, 13 Jul 2020 21:04:15 +0100 (BST) From: =?utf-8?q?Alex_Benn=C3=A9e?= To: qemu-devel@nongnu.org Subject: [PATCH v2 03/11] docker.py: fix fetching of FROM layers Date: Mon, 13 Jul 2020 21:04:07 +0100 Message-Id: <20200713200415.26214-4-alex.bennee@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200713200415.26214-1-alex.bennee@linaro.org> References: <20200713200415.26214-1-alex.bennee@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2a00:1450:4864:20::344; envelope-from=alex.bennee@linaro.org; helo=mail-wm1-x344.google.com X-detected-operating-system: by eggs.gnu.org: No matching host in p0f cache. That's all we know. X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: fam@euphon.net, berrange@redhat.com, robert.foley@linaro.org, =?utf-8?q?Alex_Benn=C3=A9e?= , richard.henderson@linaro.org, f4bug@amsat.org, robhenry@microsoft.com, =?utf-8?q?Philippe_Mathieu-Daud=C3=A9?= , aaron@os.amperecomputing.com, cota@braap.org, kuhn.chenqun@huawei.com, peter.puhov@linaro.org, aurelien@aurel32.net Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" This worked on a system that was already bootstrapped because the stage 2 images already existed even if they wouldn't be used. What we should have pulled down was the FROM line containers first because building on gitlab doesn't have the advantage of using our build system to build the pre-requisite bits. We still pull the image we want to build just in case we can use the cached data. Signed-off-by: Alex Bennée Reviewed-by: Daniel P. Berrangé Tested-by: Philippe Mathieu-Daudé --- tests/docker/docker.py | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-) -- 2.20.1 diff --git a/tests/docker/docker.py b/tests/docker/docker.py index 2d67bbd15a..c9f20d8d09 100755 --- a/tests/docker/docker.py +++ b/tests/docker/docker.py @@ -306,14 +306,18 @@ class Docker(object): checksum = _text_checksum(_dockerfile_preprocess(dockerfile)) if registry is not None: - # see if we can fetch a cache copy, may fail... - pull_args = ["pull", "%s/%s" % (registry, tag)] - if self._do(pull_args, quiet=quiet) == 0: + sources = re.findall("FROM qemu\/(.*)", dockerfile) + # Fetch any cache layers we can, may fail + for s in sources: + pull_args = ["pull", "%s/qemu/%s" % (registry, s)] + if self._do(pull_args, quiet=quiet) != 0: + registry = None + break + # Make substitutions + if registry is not None: dockerfile = dockerfile.replace("FROM qemu/", "FROM %s/qemu/" % (registry)) - else: - registry = None tmp_df = tempfile.NamedTemporaryFile(mode="w+t", encoding='utf-8', @@ -339,6 +343,8 @@ class Docker(object): build_args += ["--build-arg", "BUILDKIT_INLINE_CACHE=1"] if registry is not None: + pull_args = ["pull", "%s/%s" % (registry, tag)] + self._do(pull_args, quiet=quiet) cache = "%s/%s" % (registry, tag) build_args += ["--cache-from", cache] build_args += argv