From patchwork Thu Jan 19 18:04:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Alex_Benn=C3=A9e?= X-Patchwork-Id: 644098 Delivered-To: patch@linaro.org Received: by 2002:a17:522:b9de:b0:4b9:b062:db3b with SMTP id fj30csp359180pvb; Thu, 19 Jan 2023 10:05:05 -0800 (PST) X-Google-Smtp-Source: AMrXdXvATIVfHWdkEb5RU1tNKkV//O0tMH5A6IBMQS4s8Ty9bobHw62PZpDlgENYZpDgtGAXOsQ4 X-Received: by 2002:a05:6214:3202:b0:4d7:45fb:f5ac with SMTP id ms2-20020a056214320200b004d745fbf5acmr17465248qvb.23.1674151505747; Thu, 19 Jan 2023 10:05:05 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674151505; cv=none; d=google.com; s=arc-20160816; b=NtHGMaFhkkd7lgb6wZ6I87JqCullSCfwq5GxzWJImBoWnLGCOyMhxPzGX1BNboXofB /TBHtbL23eOEb+MrAK+XPHCsC8Tjlx8ke3PzeotZr9puqo0mpWCd0c177DyaRG+du8OG +eoSVHsMuHzYgWdmhG8VqvfuH7x9TPT8da3oC91Y2fecdFCA6H48tOlwQKQKBUS96gwM KQy3dxl271BzaflwrNa1g7uAgp2V7LfUUnYjbEN9hnnrfpLLP9L+L7P3xPz1OYYzXwml y7OLn3sUdO9ZRVIcNKDPymFGrmLDZfdNb6h8G2a87YSlAOp4Cn/xD5YTpfiqZkYQLGsr 1qyw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=bkRhIgXl5AItP6ba03zv87lTI4x9mQkrreMEEEjOaX8=; b=C1P3DwuJltKqcoR1/3zCVjXhzwQfDvLHmm6Wb4ivyw31C+QCFjMgrYgwUmFY8guTjZ 5hKQw88TzOn5KIFLlGz9yzo4nZyyfKQcniG5Xv3ed8yn+t3rgGBySV2cPVkeNu6eJVRd vWEHLTlYRxVUevMN4iE+pd5nA1ST0ItuAnszKxRZGNoziWop1cDIa+gn7iE7nnlT57Sm Pr2zybAyxv45toHX1eYDMBID+9CVyG78Fzr+fNWkzzvFAG6jO+zCVDHSQL34Mi5E9LO2 5y0OYwxFjVzhSz6FNhBFMJ2cGE6Mqk3lEg413WCP898BxHJqlWVVY9qQV9DkgnEXQr9n mjZQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=UaVie0rL; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id iy9-20020a0562140f6900b004bb607cf844si23948201qvb.134.2023.01.19.10.05.05 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Thu, 19 Jan 2023 10:05:05 -0800 (PST) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=UaVie0rL; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pIZHB-0003Ji-Uh; Thu, 19 Jan 2023 13:04:38 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pIZH2-0003Hw-LV for qemu-devel@nongnu.org; Thu, 19 Jan 2023 13:04:29 -0500 Received: from mail-wm1-x334.google.com ([2a00:1450:4864:20::334]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1pIZH0-00010v-5S for qemu-devel@nongnu.org; Thu, 19 Jan 2023 13:04:28 -0500 Received: by mail-wm1-x334.google.com with SMTP id f25-20020a1c6a19000000b003da221fbf48so1882648wmc.1 for ; Thu, 19 Jan 2023 10:04:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=bkRhIgXl5AItP6ba03zv87lTI4x9mQkrreMEEEjOaX8=; b=UaVie0rLUqSQn9ihEe0+qn9YqJxEUsAHeQ/3SjWKGQ7MXV224Mb4vN4vaEQjfZQUTx VqfK3dID9MZVUoUtdsCMBVcOpxYgRy+raCxEGtwt6AKEt9XUlUGdb6DFW18gnx5z0IGv /DM9NEMRCpyLXvVMvEH2oxZYdQh46x256Rnr5YZDSFvPSf+4WQdm0hJ7g9xtO7Dq//ft ywRmVG1m4xQb6izmsDcehiM564ODmkF0xr77OsnJ63FQ+OoyCJ5cfPHiq+Z2VOoM1Qrh yNIdPdhNrz+H7FNM5pgRGpNfHOJxtqXS4c4Vy3AZ9ImX54RhQk1cffTZpX/6gmWrbJwk R7fg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=bkRhIgXl5AItP6ba03zv87lTI4x9mQkrreMEEEjOaX8=; b=qAFwWpu8RIncoDC1TyPpV+S5IjdlrnJXbfYPuxOtNNGZMMdBasgduW42UQEUPPE74I 2uStijUPNhN0nfHGLOaONIsevhp7trUnRymRceNQourDsbY9eGKQuQagvHCuT+cQbjuz IZ4UVJ9G93aRSZuTz3cbKIr6/Cs8qFqaWSZnzfRl6+J06KiJaV18mzBE7C5ET/WoZA7w VZ0wuTDyF4juowx+0qWqvrKaoFF6nhvrUYVXt87qvMwa3hJJ24QEuk4lijwhsGO99Aq1 ZQFmg29n8U3vc4sEuweqFdt+rQ2XqTBoORcF/HbUiuzZytDziHYVmo1h7EhHnfhQd9fg D26w== X-Gm-Message-State: AFqh2kqLr911xnnUzD5CoKOyvtsHBBRGd7NhiDrJINOkRSYewvwXB+qi ep4wc6omLHJSV9vmgYJV9w6qBA== X-Received: by 2002:a05:600c:a4d:b0:3db:1d6:16f7 with SMTP id c13-20020a05600c0a4d00b003db01d616f7mr7365626wmq.23.1674151460670; Thu, 19 Jan 2023 10:04:20 -0800 (PST) Received: from zen.linaroharston ([185.81.254.11]) by smtp.gmail.com with ESMTPSA id bg14-20020a05600c3c8e00b003b4cba4ef71sm6949648wmb.41.2023.01.19.10.04.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 19 Jan 2023 10:04:20 -0800 (PST) Received: from zen.lan (localhost [127.0.0.1]) by zen.linaroharston (Postfix) with ESMTP id 919F31FFB8; Thu, 19 Jan 2023 18:04:19 +0000 (GMT) From: =?utf-8?q?Alex_Benn=C3=A9e?= To: qemu-devel@nongnu.org Cc: Beraldo Leal , Peter Maydell , Bandan Das , Qiuhao Li , =?utf-8?q?D?= =?utf-8?q?aniel_P=2E_Berrang=C3=A9?= , Michael Roth , Cleber Rosa , Alexander Bulekov , Darren Kenny , Stefan Hajnoczi , Markus Armbruster , John Snow , Ed Maste , Wainer dos Santos Moschetta , =?utf-8?q?Philippe_Mathi?= =?utf-8?q?eu-Daud=C3=A9?= , Paolo Bonzini , Li-Wen Hsu , =?utf-8?q?Marc-Andr=C3=A9_Lureau?= , =?utf-8?q?Alex_Benn=C3=A9e?= , Thomas Huth Subject: [PATCH 01/18] scripts/ci: update gitlab-runner playbook to use latest runner Date: Thu, 19 Jan 2023 18:04:02 +0000 Message-Id: <20230119180419.30304-2-alex.bennee@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230119180419.30304-1-alex.bennee@linaro.org> References: <20230119180419.30304-1-alex.bennee@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2a00:1450:4864:20::334; envelope-from=alex.bennee@linaro.org; helo=mail-wm1-x334.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: qemu-devel-bounces+patch=linaro.org@nongnu.org We were using quite and old runner on our machines and running into issues with stalling jobs. Gitlab in the meantime now reliably provide the latest packaged versions of the runner under a stable URL. This update: - creates a per-arch subdir for builds - switches from binary tarballs to deb packages - re-uses the same binary for the secondary runner - updates distro check for second to 22.04 Note this script isn't fully idempotent as we end up accumulating runners especially during testing. However we also want to be able to run twice with different GitLab keys (e.g. project and personal) so I think we just have to be mindful of that during testing. Signed-off-by: Alex Bennée --- v2 - only register aarch32 runner, move service start post both registers - tested on s390x --- scripts/ci/setup/gitlab-runner.yml | 56 +++++++----------------------- scripts/ci/setup/vars.yml.template | 2 -- 2 files changed, 13 insertions(+), 45 deletions(-) diff --git a/scripts/ci/setup/gitlab-runner.yml b/scripts/ci/setup/gitlab-runner.yml index 33128be85d..95d4199c03 100644 --- a/scripts/ci/setup/gitlab-runner.yml +++ b/scripts/ci/setup/gitlab-runner.yml @@ -50,60 +50,30 @@ - name: Download the matching gitlab-runner get_url: - dest: /usr/local/bin/gitlab-runner - url: "https://s3.amazonaws.com/gitlab-runner-downloads/v{{ gitlab_runner_version }}/binaries/gitlab-runner-{{ gitlab_runner_os }}-{{ gitlab_runner_arch }}" - owner: gitlab-runner - group: gitlab-runner - mode: u=rwx,g=rwx,o=rx - - - name: Register the gitlab-runner - command: "/usr/local/bin/gitlab-runner register --non-interactive --url {{ gitlab_runner_server_url }} --registration-token {{ gitlab_runner_registration_token }} --executor shell --tag-list {{ ansible_facts[\"architecture\"] }},{{ ansible_facts[\"distribution\"]|lower }}_{{ ansible_facts[\"distribution_version\"] }} --description '{{ ansible_facts[\"distribution\"] }} {{ ansible_facts[\"distribution_version\"] }} {{ ansible_facts[\"architecture\"] }} ({{ ansible_facts[\"os_family\"] }})'" - - - name: Install the gitlab-runner service using its own functionality - command: /usr/local/bin/gitlab-runner install --user gitlab-runner --working-directory /home/gitlab-runner - register: gitlab_runner_install_service_result - failed_when: "gitlab_runner_install_service_result.rc != 0 and \"already exists\" not in gitlab_runner_install_service_result.stderr" + dest: "/root/" + url: "https://gitlab-runner-downloads.s3.amazonaws.com/latest/deb/gitlab-runner_{{ gitlab_runner_arch }}.deb" - - name: Enable the gitlab-runner service - service: - name: gitlab-runner - state: started - enabled: yes + - name: Install gitlab-runner via package manager + apt: deb="/root/gitlab-runner_{{ gitlab_runner_arch }}.deb" - - name: Download secondary gitlab-runner - get_url: - dest: /usr/local/bin/gitlab-runner-arm - url: "https://s3.amazonaws.com/gitlab-runner-downloads/v{{ gitlab_runner_version }}/binaries/gitlab-runner-{{ gitlab_runner_os }}-arm" - owner: gitlab-runner - group: gitlab-runner - mode: u=rwx,g=rwx,o=rx - when: - - ansible_facts['distribution'] == 'Ubuntu' - - ansible_facts['architecture'] == 'aarch64' - - ansible_facts['distribution_version'] == '20.04' + - name: Register the gitlab-runner + command: "/usr/bin/gitlab-runner register --non-interactive --url {{ gitlab_runner_server_url }} --registration-token {{ gitlab_runner_registration_token }} --executor shell --tag-list {{ ansible_facts[\"architecture\"] }},{{ ansible_facts[\"distribution\"]|lower }}_{{ ansible_facts[\"distribution_version\"] }} --description '{{ ansible_facts[\"distribution\"] }} {{ ansible_facts[\"distribution_version\"] }} {{ ansible_facts[\"architecture\"] }} ({{ ansible_facts[\"os_family\"] }})'" + # The secondary runner will still run under the single gitlab-runner service - name: Register secondary gitlab-runner - command: "/usr/local/bin/gitlab-runner-arm register --non-interactive --url {{ gitlab_runner_server_url }} --registration-token {{ gitlab_runner_registration_token }} --executor shell --tag-list aarch32,{{ ansible_facts[\"distribution\"]|lower }}_{{ ansible_facts[\"distribution_version\"] }} --description '{{ ansible_facts[\"distribution\"] }} {{ ansible_facts[\"distribution_version\"] }} {{ ansible_facts[\"architecture\"] }} ({{ ansible_facts[\"os_family\"] }})'" + command: "/usr/bin/gitlab-runner register --non-interactive --url {{ gitlab_runner_server_url }} --registration-token {{ gitlab_runner_registration_token }} --executor shell --tag-list aarch32,{{ ansible_facts[\"distribution\"]|lower }}_{{ ansible_facts[\"distribution_version\"] }} --description '{{ ansible_facts[\"distribution\"] }} {{ ansible_facts[\"distribution_version\"] }} {{ ansible_facts[\"architecture\"] }} ({{ ansible_facts[\"os_family\"] }})'" when: - ansible_facts['distribution'] == 'Ubuntu' - ansible_facts['architecture'] == 'aarch64' - - ansible_facts['distribution_version'] == '20.04' + - ansible_facts['distribution_version'] == '22.04' - - name: Install the secondary gitlab-runner service using its own functionality - command: /usr/local/bin/gitlab-runner-arm install --user gitlab-runner --working-directory /home/gitlab-runner/arm -n gitlab-runner-arm + - name: Install the gitlab-runner service using its own functionality + command: "/usr/bin/gitlab-runner install --user gitlab-runner --working-directory /home/gitlab-runner" register: gitlab_runner_install_service_result failed_when: "gitlab_runner_install_service_result.rc != 0 and \"already exists\" not in gitlab_runner_install_service_result.stderr" - when: - - ansible_facts['distribution'] == 'Ubuntu' - - ansible_facts['architecture'] == 'aarch64' - - ansible_facts['distribution_version'] == '20.04' - - name: Enable the secondary gitlab-runner service + - name: Enable the gitlab-runner service service: - name: gitlab-runner-arm + name: gitlab-runner state: started enabled: yes - when: - - ansible_facts['distribution'] == 'Ubuntu' - - ansible_facts['architecture'] == 'aarch64' - - ansible_facts['distribution_version'] == '20.04' diff --git a/scripts/ci/setup/vars.yml.template b/scripts/ci/setup/vars.yml.template index e48089761f..4b355fb80f 100644 --- a/scripts/ci/setup/vars.yml.template +++ b/scripts/ci/setup/vars.yml.template @@ -1,5 +1,3 @@ -# The version of the gitlab-runner to use -gitlab_runner_version: 13.12.0 # The URL of the gitlab server to use, usually https://gitlab.com unless you're # using a private GitLab instance gitlab_runner_server_url: https://gitlab.com