From patchwork Wed Mar 20 16:13:36 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Alex_Benn=C3=A9e?= X-Patchwork-Id: 781376 Delivered-To: patch@linaro.org Received: by 2002:adf:cf01:0:b0:33e:7753:30bd with SMTP id o1csp387730wrj; Wed, 20 Mar 2024 09:14:22 -0700 (PDT) X-Forwarded-Encrypted: i=2; AJvYcCXOzDiPHRdUhgq6anK8NHEWMkxdIgb1+/8XhdgmYS9sO/a0XCF6qaCHAg9AXZScLMR0B9KT//7OzNKn5Rjkqn0x X-Google-Smtp-Source: AGHT+IEKWiMPk6U7uJ9wjWRBhw3bHi78wctpkuWOGgoEZdN5CSJqyvY3wIbnqUQ3H8oxPbn2VQ1H X-Received: by 2002:a05:620a:29d0:b0:78a:29f3:afc7 with SMTP id s16-20020a05620a29d000b0078a29f3afc7mr939173qkp.42.1710951262650; Wed, 20 Mar 2024 09:14:22 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1710951262; cv=none; d=google.com; s=arc-20160816; b=0qoTu1gEs0jmx1mpAqbzfmgK4iWF2aODHGs9n/9DmhQI/M8uDpeRBbSOfnt5B9/xUO CEGO6b3A53BzufZcoIsBdeAMt0OQyZkji86LBKJ/ZlyArhCyoeifMWh4bGWgaeIPoEgE QxbHXHRhCjR92025+QdN7PTjj4EN4TSNLgr4KQ5Sg70EsJoehrAy6iJ9G7VUyeWIeaqP qJ68Bhxn6k7Z4SaOIo1+QcLy3m1TLPgxTAANvskI/YZuI1AaqP+c9ITlV6mBUWEFjvsr pAnznFv6pU0MCC8rV69293Nbz0tOpyoRlnBH3gpjgwJLhqJRMtK3XWviVmqpANqEvE9i WhRg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=nfjqk3C9XKUuY3mZ0blhMLx9KnHWjtdF/kXD7veoPg0=; fh=0h4bzgtXQVEfkB6XdSl03pD0dH1bJixB2DvqzWWqGJM=; b=ckvWqp2YT2dmHkU0K9RjcAmz3yynWxpJmCQheExa02q3zCi8G7YUeEBzdnY8dGJx0o /jp29lvAFwGYKOFqDXX4dQvFBTXl1o/g3gmktYHtSWfnrRU4fHHsATcDcWPCAKVbiA0d PCtCT3cyxNh6zd2trfM6Hu/TdL+edkHRU2DvT+/WDpuSUYeqpTDn96hwYc4GOtMffHOG UUyz7VNZZSikB6kzEcWEWT61o8BCblh9JlrVExVJ2s+U7T0DzmC095hqtkABTPBUDqoc pDeQI8xfmq0bqqftDb+sHr2DX0GLABc5cph4+nS1qR0lrgtd+kPJJF18FyglC1czD557 m3Jg==; dara=google.com ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=NOII1P75; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id pc4-20020a05620a840400b00788258b54e3si14174373qkn.503.2024.03.20.09.14.22 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Wed, 20 Mar 2024 09:14:22 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=NOII1P75; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rmyZY-0002ze-So; Wed, 20 Mar 2024 12:13:48 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rmyZX-0002yk-A5 for qemu-devel@nongnu.org; Wed, 20 Mar 2024 12:13:47 -0400 Received: from mail-ed1-x52d.google.com ([2a00:1450:4864:20::52d]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1rmyZU-00035J-Cl for qemu-devel@nongnu.org; Wed, 20 Mar 2024 12:13:47 -0400 Received: by mail-ed1-x52d.google.com with SMTP id 4fb4d7f45d1cf-5688eaf1165so10101900a12.1 for ; Wed, 20 Mar 2024 09:13:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1710951222; x=1711556022; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=nfjqk3C9XKUuY3mZ0blhMLx9KnHWjtdF/kXD7veoPg0=; b=NOII1P754miSGkxqhcGaMg3a7KgpYSjkmzibDG9tmgdxLVqZVlCuhBRYhh2o/FhPvE MXX9EoTRgt/zr5x6WHKtRq+kI1VL20nNthtlP8HmwZt+0w2QXb8L//ueu4qpN3o7pHi7 TxKWJY7AqahWgwOz3cDH/Et8gmF9PsD2K8ElMfTXZDF63tJsUDpxucE7V4pzCCuvT/f7 dAaD+oxaL5lGMjxoMtm9odkfkJnAwY6ADCpwscLMy3TQfLczvB3vVl5B9JmguQ4YiyZT A8RNgDBNgMu4YBErlT1/YHz+FePhsEbkmra0GcE1WkKe1ET7hTb/NezNj5FSxoDv8Enx BTEQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710951222; x=1711556022; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=nfjqk3C9XKUuY3mZ0blhMLx9KnHWjtdF/kXD7veoPg0=; b=qlu9KtkdLVLUxgMIEmXJBJk1PvlBwKbsSLT7KBA30bBl70wq8q340jZGfCC+gN5anc +SurYr6ZtoVSwOQ3DSKSlx3K6JlTi/hFV+TkYyDYzF3WT5Ev8ODqjMF9x8sc7HXUt7vf qnxnS/nBasHlr+zX388S2j2BK0UNyAwEQqqk9Vyooosbxjgf0tUqDGFPzxpIuJ6qBDoV sE9LjlbJa2VpDSN6/yiZT34Yy+m6ODrs3MhFhWwmUFNc2WY05xHGfUQITvAw3xPZyIun ryaAovqbhXORMmRB3fd4UPMDWL+ZFjGyT39OUGxlPwPBj3RTO2hfvbTi+BOjhaj820Uh uslA== X-Gm-Message-State: AOJu0YxdxAZcvVxJIdHCoquvYXhGRYBmCEjk8/md7gPQCp5+DovCz3mj 1GEhP9NYx16bGwnFJ9BL9cwdDBTMJDqu8dJwBf49atBgcGcOtKeum9/APdpVF5c= X-Received: by 2002:a05:6402:528a:b0:56b:863c:2c92 with SMTP id en10-20020a056402528a00b0056b863c2c92mr4166907edb.34.1710951221848; Wed, 20 Mar 2024 09:13:41 -0700 (PDT) Received: from draig.lan ([85.9.250.243]) by smtp.gmail.com with ESMTPSA id co24-20020a0564020c1800b00568c613570dsm4682613edb.79.2024.03.20.09.13.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 20 Mar 2024 09:13:41 -0700 (PDT) Received: from draig.lan (localhost [IPv6:::1]) by draig.lan (Postfix) with ESMTP id 177FD5F8B0; Wed, 20 Mar 2024 16:13:41 +0000 (GMT) From: =?utf-8?q?Alex_Benn=C3=A9e?= To: qemu-devel@nongnu.org Cc: =?utf-8?q?Alex_Benn=C3=A9e?= , Manos Pitsidianakis , =?utf-8?q?Philippe_Ma?= =?utf-8?q?thieu-Daud=C3=A9?= , Thomas Huth , Wainer dos Santos Moschetta , Beraldo Leal , Yonggang Luo Subject: [PULL 1/5] gitlab: aggressively avoid extra GIT data Date: Wed, 20 Mar 2024 16:13:36 +0000 Message-Id: <20240320161340.391798-2-alex.bennee@linaro.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240320161340.391798-1-alex.bennee@linaro.org> References: <20240320161340.391798-1-alex.bennee@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2a00:1450:4864:20::52d; envelope-from=alex.bennee@linaro.org; helo=mail-ed1-x52d.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: qemu-devel-bounces+patch=linaro.org@nongnu.org This avoids fetching blobs and tree references for branches we are not going to worry about. Also skip tag references which are similarly not useful and keep the default --prune. This keeps the .git data to around 100M rather than the ~400M even a shallow clone takes. So we can check the savings we also run a quick du while setting up the build. We also have to have special settings of GIT_FETCH_EXTRA_FLAGS for the Windows build, the migration legacy test and the custom runners. In the case of the custom runners we also move the free floating variable to the runner template. Reviewed-by: Manos Pitsidianakis Signed-off-by: Alex Bennée Message-Id: <20240312170011.1688444-1-alex.bennee@linaro.org> diff --git a/.gitlab-ci.d/base.yml b/.gitlab-ci.d/base.yml index 2dd8a9b57c..bf3d8efab6 100644 --- a/.gitlab-ci.d/base.yml +++ b/.gitlab-ci.d/base.yml @@ -24,6 +24,10 @@ variables: # Each script line from will be in a collapsible section in the job output # and show the duration of each line. FF_SCRIPT_SECTIONS: 1 + # The project has a fairly fat GIT repo so we try and avoid bringing in things + # we don't need. The --filter options avoid blobs and tree references we aren't going to use + # and we also avoid fetching tags. + GIT_FETCH_EXTRA_FLAGS: --filter=blob:none --filter=tree:0 --no-tags --prune --quiet interruptible: true diff --git a/.gitlab-ci.d/buildtest-template.yml b/.gitlab-ci.d/buildtest-template.yml index 4fbfeb6667..22045add80 100644 --- a/.gitlab-ci.d/buildtest-template.yml +++ b/.gitlab-ci.d/buildtest-template.yml @@ -14,6 +14,7 @@ - export CCACHE_DIR="$CCACHE_BASEDIR/ccache" - export CCACHE_MAXSIZE="500M" - export PATH="$CCACHE_WRAPPERSDIR:$PATH" + - du -sh .git - mkdir build - cd build - ccache --zero-stats diff --git a/.gitlab-ci.d/buildtest.yml b/.gitlab-ci.d/buildtest.yml index c7d92fc301..cfdff175c3 100644 --- a/.gitlab-ci.d/buildtest.yml +++ b/.gitlab-ci.d/buildtest.yml @@ -187,6 +187,8 @@ build-previous-qemu: variables: IMAGE: opensuse-leap TARGETS: x86_64-softmmu aarch64-softmmu + # Override the default flags as we need more to grab the old version + GIT_FETCH_EXTRA_FLAGS: --prune --quiet before_script: - export QEMU_PREV_VERSION="$(sed 's/\([0-9.]*\)\.[0-9]*/v\1.0/' VERSION)" - git remote add upstream https://gitlab.com/qemu-project/qemu diff --git a/.gitlab-ci.d/custom-runners.yml b/.gitlab-ci.d/custom-runners.yml index 8e5b9500f4..a0e79acd39 100644 --- a/.gitlab-ci.d/custom-runners.yml +++ b/.gitlab-ci.d/custom-runners.yml @@ -10,13 +10,14 @@ # gitlab-runner. To avoid problems that gitlab-runner can cause while # reusing the GIT repository, let's enable the clone strategy, which # guarantees a fresh repository on each job run. -variables: - GIT_STRATEGY: clone # All custom runners can extend this template to upload the testlog # data as an artifact and also feed the junit report .custom_runner_template: extends: .base_job_template + variables: + GIT_STRATEGY: clone + GIT_FETCH_EXTRA_FLAGS: --no-tags --prune --quiet artifacts: name: "$CI_JOB_NAME-$CI_COMMIT_REF_SLUG" expire_in: 7 days diff --git a/.gitlab-ci.d/windows.yml b/.gitlab-ci.d/windows.yml index f116b8012d..94834269ec 100644 --- a/.gitlab-ci.d/windows.yml +++ b/.gitlab-ci.d/windows.yml @@ -28,6 +28,8 @@ msys2-64bit: # qTests don't run successfully with "--without-default-devices", # so let's exclude the qtests from CI for now. TEST_ARGS: --no-suite qtest + # The Windows git is a bit older so override the default + GIT_FETCH_EXTRA_FLAGS: --no-tags --prune --quiet artifacts: name: "$CI_JOB_NAME-$CI_COMMIT_REF_SLUG" expire_in: 7 days