From patchwork Wed Jan 3 17:33:26 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Alex_Benn=C3=A9e?= X-Patchwork-Id: 759636 Delivered-To: patch@linaro.org Received: by 2002:a5d:67c6:0:b0:336:6142:bf13 with SMTP id n6csp6612434wrw; Wed, 3 Jan 2024 09:40:46 -0800 (PST) X-Google-Smtp-Source: AGHT+IH4N1mKPyPw6vZzNTGWtvCkvQF2nBSD/Hx/n+h5tGxUeJQje2Zgh4OzR+C6A2nQTVPV+Xeh X-Received: by 2002:a05:6808:2388:b0:3bc:26c4:fc99 with SMTP id bp8-20020a056808238800b003bc26c4fc99mr1324100oib.54.1704303646301; Wed, 03 Jan 2024 09:40:46 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1704303646; cv=none; d=google.com; s=arc-20160816; b=WUyG93v3IxWdnou5tJDWxBJWQcrv9984tVzM+3EyydPXFPnzQJJ8TG7zxe6KS+6q9m npD6JccGEw4it93P/rl6WQMtF0YoIBl7waeU7IFMYROg7e1sztru0z+gwryVoVqK3+z5 oAK8RvLK8oe4UgJWokd4rgSt486B+10JWaBF2R5OeJJ5es9iGSdk8zuH0lq02F/dgoRE OJcdJDp+EIAJZyV5sxLbBn/BYJV/WU31MnE8AqUoNvWgJBCUWpIyipU+zq2AJiNgz4ST Ui+mJFHIorqqojt5xU8cpaKwZ/OkQSfwJPeMc+cDYSpv3/qt25RgFm9USZWkJJuT27me KNOQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=0sLk9Y6V09YT5mxoKGySQqDe7AUl7PrL/RJQ+Y1wnbw=; fh=vgfOyrYvOwgmYzstDn0/HgLmIVrGMIP63mUycC+6pY8=; b=DbYaZh2m53N0Iq5y1TS69bxD6LrNWIopbqGQrLh7yulAkWL2qH+CaUFVmGvSKnDt9t CuqNCrYTyaPKGo1XQwe/SdcE5xtFMunHIiq9G8E9bbWdVIPpnCf/jBmlhk3A5xTEhka7 rZNrmoYZtgyPcquqO2elHtOsTgXyCxVfRiUzn+p8w+/svKUT3IMtov8XiEiGAM6owZ4c xShIfHv6Z99O8E/t+azgm9aK94pb83Qqx7bHWGpWw+hLnAf7TFudrKeVp8hWErEJEBVi 77nDwBSToXZVPkrS8KUOFNHc8p8lweA0eOT3UwNxkdHpQqOaYKYkIRgNSaEScU1n3aSD 8tvQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=gxIsOFzW; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id b13-20020a0cc98d000000b0067f3089e006si31093123qvk.546.2024.01.03.09.40.45 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Wed, 03 Jan 2024 09:40:46 -0800 (PST) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=gxIsOFzW; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rL5DG-0006v9-LY; Wed, 03 Jan 2024 12:39:30 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rL5D1-00068q-Vo for qemu-devel@nongnu.org; Wed, 03 Jan 2024 12:39:15 -0500 Received: from mail-wm1-x32d.google.com ([2a00:1450:4864:20::32d]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1rL5Cu-0000c2-Oq for qemu-devel@nongnu.org; Wed, 03 Jan 2024 12:39:15 -0500 Received: by mail-wm1-x32d.google.com with SMTP id 5b1f17b1804b1-40d604b4b30so4368485e9.1 for ; Wed, 03 Jan 2024 09:39:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1704303546; x=1704908346; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=0sLk9Y6V09YT5mxoKGySQqDe7AUl7PrL/RJQ+Y1wnbw=; b=gxIsOFzWh6zn0hez+uEbsJ5bUiUqIo6ErnyzmnBKsBWFd4K9p1Azi4Lr2Qru3tc7C5 u5RvlrXh3WukJLnbTTUORACUx4ZsZ9zKjiB4BBT6AEVLmb4g7bOz8+COgSmLAMO0ZhYg 7zrpCmH75KpJlyU0rbKP9s15GXsPJmCVlrEXjO1f6d6YXc0nxTlrmIeFnjDkNWOUgTQD ybkNmgu9LvLFEjHTub642lxMvOREptjiC0wkDBSalCaDSQxxnSdS99Ui8AUi9uwkQ3kD eH1HB/n4sapgFGrgNXItj6By6ZyLsY8rgnwYvNRap+Nciq8PRbD/INszUj2ejGsGnVYf /6yQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1704303546; x=1704908346; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=0sLk9Y6V09YT5mxoKGySQqDe7AUl7PrL/RJQ+Y1wnbw=; b=ZefWpLhGzVbOylYWidl1NUDavT8PSfybdafHznQCp3jTR+fx+hKR4fJhm30GK2a2q3 QvVLzmp7aRW98I6zHoT2NcjvyJs1wlMVMEBIdbQZl7BH1PQbnFbspVe5JgV0eR7NTk83 MT2iKbOgJ6rcsRGv0b5zqwie5XU/z/dZp5JMTWhgFgVhtq1w5Fzd99NeZcVCS1u/Ul4c JDNxEVg28TMFFXH9KiMZIjypXNeYJmdJnP57mCuk2DuxIZ67xbKBdSt5tHBpVzgXjjVX SJpeLBpRbUO5K8WUpjl9+6av1hCR+3O66wZDZAo+/dVKls4yJP6XS2soKLsWshPkKeOZ T3IQ== X-Gm-Message-State: AOJu0YxihnlolCQZwGmWcHusmwQowDXy6Du9u0aI2NUOsgndSRRu1G+c +f42+BtJvBkEUhlJYYJH5MaXzhcKL8kbiA== X-Received: by 2002:a05:600c:4fc7:b0:40d:3f30:a09e with SMTP id o7-20020a05600c4fc700b0040d3f30a09emr804540wmq.84.1704303546006; Wed, 03 Jan 2024 09:39:06 -0800 (PST) Received: from draig.lan ([85.9.250.243]) by smtp.gmail.com with ESMTPSA id dr16-20020a5d5f90000000b003373ef060d5sm8567186wrb.113.2024.01.03.09.39.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 03 Jan 2024 09:39:05 -0800 (PST) Received: from draig.lan (localhost [IPv6:::1]) by draig.lan (Postfix) with ESMTP id 298A25F950; Wed, 3 Jan 2024 17:33:51 +0000 (GMT) From: =?utf-8?q?Alex_Benn=C3=A9e?= To: qemu-devel@nongnu.org Cc: qemu-s390x@nongnu.org, qemu-ppc@nongnu.org, Richard Henderson , Song Gao , =?utf-8?q?Marc-Andr=C3=A9_Lureau?= , David Hildenbrand , Aurelien Jarno , Yoshinori Sato , Yanan Wang , Bin Meng , Laurent Vivier , Michael Rolnik , Alexandre Iooss , David Woodhouse , Laurent Vivier , Paolo Bonzini , Brian Cain , Daniel Henrique Barboza , Beraldo Leal , Paul Durrant , Mahmoud Mandour , Thomas Huth , Liu Zhiwei , Cleber Rosa , kvm@vger.kernel.org, Peter Maydell , Wainer dos Santos Moschetta , =?utf-8?q?Alex_Benn?= =?utf-8?q?=C3=A9e?= , qemu-arm@nongnu.org, Weiwei Li , =?utf-8?q?Philippe_Mathieu-Daud=C3=A9?= , John Snow , Daniel Henrique Barboza , Nicholas Piggin , Palmer Dabbelt , Marcel Apfelbaum , Ilya Leoshkevich , =?utf-8?q?C=C3=A9dric_Le_Goater?= , "Edgar E. Iglesias" , Eduardo Habkost , Pierrick Bouvier , qemu-riscv@nongnu.org, Alistair Francis , =?utf-8?q?Daniel_P=2E_Berrang?= =?utf-8?q?=C3=A9?= Subject: [PATCH v2 20/43] mtest2make: stop disabling meson test timeouts Date: Wed, 3 Jan 2024 17:33:26 +0000 Message-Id: <20240103173349.398526-21-alex.bennee@linaro.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240103173349.398526-1-alex.bennee@linaro.org> References: <20240103173349.398526-1-alex.bennee@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2a00:1450:4864:20::32d; envelope-from=alex.bennee@linaro.org; helo=mail-wm1-x32d.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: qemu-devel-bounces+patch=linaro.org@nongnu.org From: Daniel P. Berrangé The mtest2make.py script passes the arg '-t 0' to 'meson test' which disables all test timeouts. This is a major source of pain when running in GitLab CI and a test gets stuck. It will stall until GitLab kills the CI job. This leaves us with little easily consumable information about the stalled test. The TAP format doesn't show the test name until it is completed, and TAP output from multiple tests it interleaved. So we have to analyse the log to figure out what tests had un-finished TAP output present and thus infer which test case caused the hang. This is very time consuming and error prone. By allowing meson to kill stalled tests, we get a direct display of what test program got stuck, which lets us more directly focus in on what specific test case within the test program hung. The other issue with disabling meson test timeouts by default is that it makes it more likely that maintainers inadvertantly introduce slowdowns. For example the recent-ish change that accidentally made migrate-test take 15-20 minutes instead of around 1 minute. The main risk of this change is that the individual test timeouts might be too short to allow completion in high load scenarios. Thus, there is likely to be some short term pain where we have to bump the timeouts for certain tests to make them reliable enough. The preceeding few patches raised the timeouts for all failures that were immediately apparent in GitLab CI. Even with the possible short term instability, this should still be a net win for debuggability of failed CI pipelines over the long term. Signed-off-by: Daniel P. Berrangé Reviewed-by: Philippe Mathieu-Daudé Message-ID: <20230717182859.707658-13-berrange@redhat.com> Signed-off-by: Thomas Huth Message-Id: <20231215070357.10888-17-thuth@redhat.com> Signed-off-by: Alex Bennée --- scripts/mtest2make.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/scripts/mtest2make.py b/scripts/mtest2make.py index 179dd548718..eb01a05ddbd 100644 --- a/scripts/mtest2make.py +++ b/scripts/mtest2make.py @@ -27,7 +27,8 @@ def names(self, base): .speed.slow = $(foreach s,$(sort $(filter-out %-thorough, $1)), --suite $s) .speed.thorough = $(foreach s,$(sort $1), --suite $s) -.mtestargs = --no-rebuild -t 0 +TIMEOUT_MULTIPLIER = 1 +.mtestargs = --no-rebuild -t $(TIMEOUT_MULTIPLIER) ifneq ($(SPEED), quick) .mtestargs += --setup $(SPEED) endif