From patchwork Wed Nov 1 10:28:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maxim Kuvyrkov X-Patchwork-Id: 740058 Delivered-To: patch@linaro.org Received: by 2002:a5d:538f:0:b0:32d:baff:b0ca with SMTP id d15csp205516wrv; Wed, 1 Nov 2023 03:28:53 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHlcrcM81uG8CLIscOGMguiW4isXLGNp2onBfpEKbPfIPZknSiPTishQRBIFnXDwESgxAyv X-Received: by 2002:a05:622a:1914:b0:408:392e:2aa5 with SMTP id w20-20020a05622a191400b00408392e2aa5mr3180649qtc.20.1698834533447; Wed, 01 Nov 2023 03:28:53 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1698834533; cv=pass; d=google.com; s=arc-20160816; b=X982lVWy3IBu1YOmMBXg2aow+q6UsVTNTkxP9iDVe+LYeqqv2YsIaTadnGVrAUunCM 2eNcSCdFNm9Zw3g9rWOFSk8msIwQHzN1Asp3d3mZrDgeX3ALwF0P0cwONm6sZNJiYfuA TiysyplJRGTwDPOzDTBeBNyg9uahesCdstuDoY1MSEC51UoAHFXax6VeiT72rT6pdvdt YH37H7bGnGiLT5NbqXZJdxTB1mi1IcziujwYR5BBwGWWJOlE/a6s7cSAD7b+YH02bbP4 YJWt1+YRnqn6C6fleQFrePyntEOThSMwk2autrfuG0jR19U/8jS8K0F4Y36gMJFZdovK wZ0w== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature:arc-filter:dmarc-filter:delivered-to; bh=HR6Y/gQA2x6lN25nILPzoSYcKFQvOsy8sazIc700DqU=; fh=toLv9C89QaKLq6B/p9fNKWFWc40r530S2VrtIJtA0ks=; b=rH84MdK5N+Csptfvpby0rSeVeX8r3tzGpfSP+kgsUjujCLLNGANyj9qeVPzd8SCcD5 GqfNPzc4TtVbqsxsni4WgzmjIsto2jfkrVZXnlFrmGn4QC17R+DcMdrfO1utNljh8xKt Bb4WiIsxiZAx1YI2ots9izYe5gKpeK99OOsGtaX/gZWayr7VRGo3G+UONQhDbULxFdsZ m2eBW807nJrCuOK0l7rmDy+YO3LEw/QOm/FT80KxWgEv7wqlszXLq5xwfc0ag4J7XkbA 9mula5M9OOK9DfH/1TyHPi1wsleb/bSsQRsTdMnyTe1Wl9WxzyFJH2Jve7ZMvFi9h2Rf jbqQ== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b="jigb/30S"; arc=pass (i=1); spf=pass (google.com: domain of libc-alpha-bounces+patch=linaro.org@sourceware.org designates 2620:52:3:1:0:246e:9693:128c as permitted sender) smtp.mailfrom="libc-alpha-bounces+patch=linaro.org@sourceware.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from server2.sourceware.org (server2.sourceware.org. [2620:52:3:1:0:246e:9693:128c]) by mx.google.com with ESMTPS id s2-20020a05622a1a8200b0041814373040si2567430qtc.49.2023.11.01.03.28.53 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 01 Nov 2023 03:28:53 -0700 (PDT) Received-SPF: pass (google.com: domain of libc-alpha-bounces+patch=linaro.org@sourceware.org designates 2620:52:3:1:0:246e:9693:128c as permitted sender) client-ip=2620:52:3:1:0:246e:9693:128c; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b="jigb/30S"; arc=pass (i=1); spf=pass (google.com: domain of libc-alpha-bounces+patch=linaro.org@sourceware.org designates 2620:52:3:1:0:246e:9693:128c as permitted sender) smtp.mailfrom="libc-alpha-bounces+patch=linaro.org@sourceware.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 1E0763856965 for ; Wed, 1 Nov 2023 10:28:53 +0000 (GMT) X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from mail-lf1-x12c.google.com (mail-lf1-x12c.google.com [IPv6:2a00:1450:4864:20::12c]) by sourceware.org (Postfix) with ESMTPS id 234D63858C00 for ; Wed, 1 Nov 2023 10:28:36 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 234D63858C00 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=linaro.org Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=linaro.org ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 234D63858C00 Authentication-Results: server2.sourceware.org; arc=none smtp.remote-ip=2a00:1450:4864:20::12c ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1698834526; cv=none; b=sQdLBP0Q8qc37aSHMd0zV9ooLAQyBOszMkixkfZxpDelVUm7ZnADjuxRLQoCWGvEZD7D5cRHpqeAVVX3ldts07VzTFvLrFD5aaV/aTbZ7+76X8jB0FKPh6TFzSl66Y54gCQkp2R5zGBN5mhm8UrQ9DZqgg01tiC+P4nrapLrpGw= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1698834526; c=relaxed/simple; bh=DLz1XLXudrdhTI1pmIwHDNHmOYcm+2llMf8f1oEahFQ=; h=DKIM-Signature:From:To:Subject:Date:Message-Id:MIME-Version; b=drJHoNiWgM5JmUA61NQmDrdccO17kM+Vi3Y90f3xcIaZ1iyMwEkTg+T5AQaaPrOTYmv6KmLF00lTSXCW7hTNzxxUCCueZN5TKUy/MB4Z+Uk3xCE8/hHo66zo3CfE/QXesvzjyVkj6GdfhpnJuwBlFOn6L9tXyDj52HMMeFT1dUk= ARC-Authentication-Results: i=1; server2.sourceware.org Received: by mail-lf1-x12c.google.com with SMTP id 2adb3069b0e04-507f63fcbf6so2230719e87.1 for ; Wed, 01 Nov 2023 03:28:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1698834514; x=1699439314; darn=sourceware.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=HR6Y/gQA2x6lN25nILPzoSYcKFQvOsy8sazIc700DqU=; b=jigb/30Scb8QFI/7aznrmyGIzsFccM/hRALJ3Bn/1H+OUj8pe0bWoBKUxVheAMCk7y xV5M1ZCssjSgaV6xbkOxn0N2l3OCeARos8lj/c6TltaaXRCS3aOhlHm/+G0jdLwS5Z3c Tbhx6aOrji+xXveeUZRMxHKTj99aYP0/8tKVOtANn5p+rYjnjDlK/UtjrF8vyFANpCEx rdBmbb7u+jsomSnoux2RT4rJnQvewiC74D5BL7qSKNtchU5DoPQJOnYQ+F2PKLDWe5PC 1Q13p/qQ//V6pEJQ0MKOt8vhykrcH4Mfh3CGPPNhUi+j2EeUGK+/EmnJRj2Lxa7hA7v5 eC7Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698834514; x=1699439314; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=HR6Y/gQA2x6lN25nILPzoSYcKFQvOsy8sazIc700DqU=; b=BOn1KZuiFsntpOad8/vukJ9AhTnN5LN27hzUXB8qTmAU2WtHPGboyCWQXKdk6X1grk LQXbvH55upBCa+ZcVYmti9KL7uRM/gv1AxIhWPpmWUNuxClLcUNLXOf7cQYZVCtxXmKy pGTIHJF6q4tiBKn/hl38VS71coUJ4yTjz3DPe7JLgOKTHvZmVaKk44UmZ8n4kAjGSxuB u9uu8uTAGnGZXJXuKJiyFQOlg7/4EKHtHtitcc5Npz9GVkhEI1zkUH+z3nypTMu4Qkbs iHoGU6R+V9M07zEeHnied8xFcKu+fQlE6HEKvkbCwkFFm5EtlZQK6CRW8ZcbM8f8s+LP tdGg== X-Gm-Message-State: AOJu0YxMMYpKa0RZo5ZMBtz5c2mNJsnRnN8UoWbQMjXMsrXDcBhhU7tT sbiL10DACXgHyj58H7ao7G3YlxqI0/q9etldphQm X-Received: by 2002:a05:6512:1242:b0:507:9d5c:62e3 with SMTP id fb2-20020a056512124200b005079d5c62e3mr11736334lfb.5.1698834513851; Wed, 01 Nov 2023 03:28:33 -0700 (PDT) Received: from localhost.localdomain (static.225.72.216.95.clients.your-server.de. [95.216.72.225]) by smtp.gmail.com with ESMTPSA id w13-20020a0565120b0d00b00502ae8db086sm181031lfu.19.2023.11.01.03.28.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 01 Nov 2023 03:28:32 -0700 (PDT) From: Maxim Kuvyrkov To: libc-alpha@sourceware.org Cc: Maxim Kuvyrkov , Florian Weimer Subject: [PATCH v2] Format test results closer to what DejaGnu does Date: Wed, 1 Nov 2023 10:28:28 +0000 Message-Id: <20231101102828.3207231-1-maxim.kuvyrkov@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230601101424.1848817-1-maxim.kuvyrkov@linaro.org> References: <20230601101424.1848817-1-maxim.kuvyrkov@linaro.org> MIME-Version: 1.0 X-Spam-Status: No, score=-12.0 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libc-alpha-bounces+patch=linaro.org@sourceware.org The only change from v1 is removal of "=== glibc failures ===" line. The years of dealing with Binutils, GCC and GDB test results made the community create good tools for comparison and analysis of DejaGnu test results. This change allows to use those tools for Glibc's test results as well. The motivation for this change is Linaro's pre-commit testers, which use a modified version of GCC's validate_failures.py to create test xfail lists with baseline failures and known flaky tests. See below links for an example xfails file (only one link is supposed to work at any given time): - https://ci.linaro.org/job/tcwg_glibc_check--master-arm-build/lastSuccessfulBuild/artifact/artifacts/artifacts.precommit/sumfiles/xfails.xfail/*view*/ - https://ci.linaro.org/job/tcwg_glibc_check--master-arm-build/lastSuccessfulBuild/artifact/artifacts/sumfiles/xfails.xfail/*view*/ Specifacally, this patch changes format of glibc's .sum files from ... FAIL: elf/test1 PASS: string/test2 ... to ... === glibc tests === Running elf ... FAIL: elf/test1 Running string ... PASS: string/test2 . And output of "make check" from ... FAIL: elf/test1 ... to ... FAIL: elf/test1 === Summary of results === 1 FAIL 1 PASS . Signed-off-by: Maxim Kuvyrkov Reviewed-by: Adhemerval Zanella --- Makefile | 11 +++++++---- scripts/merge-test-results.sh | 3 +++ 2 files changed, 10 insertions(+), 4 deletions(-) diff --git a/Makefile b/Makefile index b938721166..a4f3378e21 100644 --- a/Makefile +++ b/Makefile @@ -577,11 +577,14 @@ $(objpfx)lint-makefiles.out: scripts/lint-makefiles.sh $(SHELL) $< "$(PYTHON)" `pwd` > $@ ; \ $(evaluate-test) +# Print test summary for tests in $1 .sum file; +# $2 is optional test identifier. +# Fail if there are unexpected failures in the test results. define summarize-tests -@grep -E -v '^(PASS|XFAIL):' $(objpfx)$1 || true -@echo "Summary of test results$2:" -@sed 's/:.*//' < $(objpfx)$1 | sort | uniq -c -@! grep -E -q -v '^(X?PASS|XFAIL|UNSUPPORTED):' $(objpfx)$1 +@grep -E '^[A-Z]+:' $(objpfx)$1 | grep -E -v '^(PASS|XFAIL):' || true +@echo " === Summary of results$2 ===" +@sed -e '/:.*/!d' -e 's/:.*//' < $(objpfx)$1 | sort | uniq -c +@! grep -E '^[A-Z]+:' $(objpfx)$1 | grep -E -q -v '^(X?PASS|XFAIL|UNSUPPORTED):' endef # The intention here is to do ONE install of our build into the diff --git a/scripts/merge-test-results.sh b/scripts/merge-test-results.sh index e4dcc2520a..8f9d81f6eb 100755 --- a/scripts/merge-test-results.sh +++ b/scripts/merge-test-results.sh @@ -50,7 +50,10 @@ case $type in -t) subdir_file_name=$1 shift + echo " === glibc tests ===" for d in "$@"; do + echo + echo "Running $d ..." if [ -f "$objpfx$d/$subdir_file_name" ]; then cat "$objpfx$d/$subdir_file_name" else