From patchwork Thu Dec 21 10:37:58 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Alex_Benn=C3=A9e?= X-Patchwork-Id: 757013 Delivered-To: patch@linaro.org Received: by 2002:a5d:67c6:0:b0:336:6142:bf13 with SMTP id n6csp818759wrw; Thu, 21 Dec 2023 02:51:04 -0800 (PST) X-Google-Smtp-Source: AGHT+IG/dMlCYlnJ9qhFzv3QA/sQHm6OU0tG0PLeUxhNsOivnPTCb0jEbeBeisjGN0SriO72zgvV X-Received: by 2002:a05:6808:38c7:b0:3ba:40de:a45d with SMTP id el7-20020a05680838c700b003ba40dea45dmr8789201oib.111.1703155864311; Thu, 21 Dec 2023 02:51:04 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1703155864; cv=none; d=google.com; s=arc-20160816; b=Ipe287JlHBpUbqbqrk/aEJ9fteEKFn5nb3hCMAl9FI80mDpvUaCkpJxoBiRUZgEMv+ tWek2KYvxAWpJqS91ENfvvuIpQL++/3M9hgXU/Po5Mamx8+fjTp9mDkHGuzmTlT+DzyU PfGssgQpdWt0VJoCQfI+IBi6vMq2nd7UO6L/mZijva/amRKgaCEC96nbJI2rqzijfP+2 H4KtpFrU426SykmFxtG8sNXTu5NW/Bgqgu4opBtw9vI8gShGU3EDftcFNnlxptyqqBd+ Wruo/L9LOQ33ZPxtn7MdhK+W+VMX42N2ljd0C2BM1ogkzP6rgIWWts4UbUwuurQVm7eA jJPQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=0sLk9Y6V09YT5mxoKGySQqDe7AUl7PrL/RJQ+Y1wnbw=; fh=iW9PmIx3/QjmF+pbeEystwBNgx6qY50iYwsybT0SqaQ=; b=HMx3YLrI0T1B3Sv6pXRRbkFkNw8Vx4M21uJimRhbieiMzsVTb5ozL0up9KAPmpzDmV sSxpMCOZaS97VfAxnCoOEYQkBWyQhcUacm9TVdovQfIo6BbJGzIrrhwcdiJjxQenmVdz pUW4LwlxBtfulDYaX5LMDvzC6Goxcv/qLzqG/jTe44/752EDVIzNOD4394feYX2t7Xvf 8d2yBkrr4eT+sgiK9gRuTPQmEI8CUBl1QhmZCsotF/sfOpZVqEc1ocCJgQpd55FMWhtW xpFhT+NsfF54PhLjpXg6+Byzt89yMopU6p0v0/856LwvrChh+hUHe1TnDKFb0bnBmbfW pLhA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b="l/GHXLTi"; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id n5-20020a0c9d45000000b0067f2443e575si1890272qvf.165.2023.12.21.02.51.04 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Thu, 21 Dec 2023 02:51:04 -0800 (PST) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b="l/GHXLTi"; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rGGap-0004pr-Hj; Thu, 21 Dec 2023 05:47:55 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rGGae-0004Yk-Sp for qemu-devel@nongnu.org; Thu, 21 Dec 2023 05:47:44 -0500 Received: from mail-wr1-x429.google.com ([2a00:1450:4864:20::429]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1rGGa8-0007NF-Ol for qemu-devel@nongnu.org; Thu, 21 Dec 2023 05:47:44 -0500 Received: by mail-wr1-x429.google.com with SMTP id ffacd0b85a97d-336755f1688so560125f8f.0 for ; Thu, 21 Dec 2023 02:47:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1703155627; x=1703760427; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=0sLk9Y6V09YT5mxoKGySQqDe7AUl7PrL/RJQ+Y1wnbw=; b=l/GHXLTi5BaMXE3URvo3WSFhVISs5dabLukqHCPcjioGnzOBC3N1x87ivjLk1ovitt H8MKDBz8GnqOIttYAvS79cOyo1gY8ccQ52SxwyHUde7JvjG/Rf8mgdXY4KD0gV8BgC0O 4RIObDBZ0OztL9ecfwdk6KY9rJpf32zk866BquXZX3GIh/LIwzG3/G1fDzFGbymvv3rI 7DJeo43/ptSLTOn7SL8X9APgIHzgp9SIkQx+XeIq8eTwIf2G0bPtz+MIOmKAb1VHn48s PYa4RShlbJ0ph72XKiABAbdSfSOsAqvtaAj5un3UVL07Qn9WJFL5XvuPmgngpKSjMZqp mrJg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1703155627; x=1703760427; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=0sLk9Y6V09YT5mxoKGySQqDe7AUl7PrL/RJQ+Y1wnbw=; b=l5pXIfuIPUptEqfYk0D7u7xJWH5e6/YWS4befl8DZXH1gDSQfRkQ0qiC9K1QtNEOVD wPCPWA5+9+Bi1kX6rLhWalHpAyI/KrH0bPANTgUdjaDiPvYiWZJr7Ie9uhNzW9PwVp0l biPEZ09+Kk4JqM2bEdSZlJZwFECOr1Wd9Ah5A8YVUozhfBb4JRxfkex5s5o7oYC1rYLW We2amnBzdepTh4EqC0dkyF59Vul/tdhuoxt6ttnhDfy/QvZo3pAMWkfnGBuPKfd0z0Wn w5AuQYogSRsHY9mi14LJ0JyN9ulXAhGFHIh20g9HofNPcjqVl709da798UT78N05xWZl xEbA== X-Gm-Message-State: AOJu0YyrQGKDRIJ9eBa7bGItP9wbHRDkpAAOti37Bq1RD3PX1N38ZvvL FltygkcdasnRfgMPhw2/AfFK/g== X-Received: by 2002:a05:600c:1682:b0:40c:3984:4977 with SMTP id k2-20020a05600c168200b0040c39844977mr650705wmn.174.1703155627469; Thu, 21 Dec 2023 02:47:07 -0800 (PST) Received: from draig.lan ([85.9.250.243]) by smtp.gmail.com with ESMTPSA id b2-20020a05600c4e0200b0040c440f9393sm3526519wmq.42.2023.12.21.02.47.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 21 Dec 2023 02:47:06 -0800 (PST) Received: from draig.lan (localhost [IPv6:::1]) by draig.lan (Postfix) with ESMTP id CEF355F8E3; Thu, 21 Dec 2023 10:38:20 +0000 (GMT) From: =?utf-8?q?Alex_Benn=C3=A9e?= To: qemu-devel@nongnu.org Cc: "Edgar E. Iglesias" , John Snow , Aurelien Jarno , =?utf-8?q?Philippe_Mathieu-Daud=C3=A9?= , Yanan Wang , Eduardo Habkost , Brian Cain , Laurent Vivier , Palmer Dabbelt , Cleber Rosa , David Hildenbrand , Beraldo Leal , Pierrick Bouvier , Weiwei Li , =?utf-8?q?Alex_Benn=C3=A9e?= , Paul Durrant , qemu-s390x@nongnu.org, David Woodhouse , Liu Zhiwei , Ilya Leoshkevich , Wainer dos Santos Moschetta , Michael Rolnik , Alistair Francis , Daniel Henrique Barboza , Laurent Vivier , kvm@vger.kernel.org, =?utf-8?q?Marc-An?= =?utf-8?q?dr=C3=A9_Lureau?= , Alexandre Iooss , Thomas Huth , Peter Maydell , qemu-ppc@nongnu.org, Paolo Bonzini , Marcel Apfelbaum , Nicholas Piggin , qemu-riscv@nongnu.org, qemu-arm@nongnu.org, Song Gao , Yoshinori Sato , Richard Henderson , Daniel Henrique Barboza , =?utf-8?q?C=C3=A9dric_?= =?utf-8?q?Le_Goater?= , Mahmoud Mandour , Bin Meng , =?utf-8?q?Daniel_P=2E_Berrang=C3=A9?= Subject: [PATCH 20/40] mtest2make: stop disabling meson test timeouts Date: Thu, 21 Dec 2023 10:37:58 +0000 Message-Id: <20231221103818.1633766-21-alex.bennee@linaro.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20231221103818.1633766-1-alex.bennee@linaro.org> References: <20231221103818.1633766-1-alex.bennee@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2a00:1450:4864:20::429; envelope-from=alex.bennee@linaro.org; helo=mail-wr1-x429.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: qemu-devel-bounces+patch=linaro.org@nongnu.org From: Daniel P. Berrangé The mtest2make.py script passes the arg '-t 0' to 'meson test' which disables all test timeouts. This is a major source of pain when running in GitLab CI and a test gets stuck. It will stall until GitLab kills the CI job. This leaves us with little easily consumable information about the stalled test. The TAP format doesn't show the test name until it is completed, and TAP output from multiple tests it interleaved. So we have to analyse the log to figure out what tests had un-finished TAP output present and thus infer which test case caused the hang. This is very time consuming and error prone. By allowing meson to kill stalled tests, we get a direct display of what test program got stuck, which lets us more directly focus in on what specific test case within the test program hung. The other issue with disabling meson test timeouts by default is that it makes it more likely that maintainers inadvertantly introduce slowdowns. For example the recent-ish change that accidentally made migrate-test take 15-20 minutes instead of around 1 minute. The main risk of this change is that the individual test timeouts might be too short to allow completion in high load scenarios. Thus, there is likely to be some short term pain where we have to bump the timeouts for certain tests to make them reliable enough. The preceeding few patches raised the timeouts for all failures that were immediately apparent in GitLab CI. Even with the possible short term instability, this should still be a net win for debuggability of failed CI pipelines over the long term. Signed-off-by: Daniel P. Berrangé Reviewed-by: Philippe Mathieu-Daudé Message-ID: <20230717182859.707658-13-berrange@redhat.com> Signed-off-by: Thomas Huth Message-Id: <20231215070357.10888-17-thuth@redhat.com> Signed-off-by: Alex Bennée --- scripts/mtest2make.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/scripts/mtest2make.py b/scripts/mtest2make.py index 179dd548718..eb01a05ddbd 100644 --- a/scripts/mtest2make.py +++ b/scripts/mtest2make.py @@ -27,7 +27,8 @@ def names(self, base): .speed.slow = $(foreach s,$(sort $(filter-out %-thorough, $1)), --suite $s) .speed.thorough = $(foreach s,$(sort $1), --suite $s) -.mtestargs = --no-rebuild -t 0 +TIMEOUT_MULTIPLIER = 1 +.mtestargs = --no-rebuild -t $(TIMEOUT_MULTIPLIER) ifneq ($(SPEED), quick) .mtestargs += --setup $(SPEED) endif