From patchwork Fri May 25 14:19:15 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Alex_Benn=C3=A9e?= X-Patchwork-Id: 136918 Delivered-To: patch@linaro.org Received: by 2002:a2e:9706:0:0:0:0:0 with SMTP id r6-v6csp3670205lji; Fri, 25 May 2018 07:34:04 -0700 (PDT) X-Google-Smtp-Source: ADUXVKK9s1F8S7I5aLW6/ddaTAHu4KQTX07dslNe1Yd8+4mlrQB0z35tjDXcUKiLzAbMYaGalsQe X-Received: by 2002:a37:3c45:: with SMTP id j66-v6mr2276804qka.328.1527258843915; Fri, 25 May 2018 07:34:03 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1527258843; cv=none; d=google.com; s=arc-20160816; b=faUYNhz+5e7KrFU/tZgzzMJZXARS/Hd5rbBaI6pXiANMWFGodyUqAmUe6ef9pNX69U /xteUvHcVI01E6+SNecnHhhiDveDl1Muz4QoPZqFo3E4D22eSq5Aw3T6J1Cwno/pTNcP b400dYUNo/2Ia1VpvDnTec4md3YiymISDnjMKzt0er9P+yHAX87q+Cf1VCkPjavCy67N abiAXfane8YTyBY9uV2BiB2J+6iKeQTWolu4upy520sdKmqJZJ2gaRrBH0oQGHVSltfy Q1XK4OUTPHNHUsVhsODQodkfhkguxLyzvUzJzKlUmeo6Av5pGltDRpr8LuhTSuJOaNcA z9bw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:cc:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject :content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:arc-authentication-results; bh=Ilb+xV/1jTS1+/Rs3a4Ap9VqiRVoeMi1JNESVSUvGu8=; b=U5BmrXZa4U9oDZetnIS3GkVzCmdl4lfCC7OgHc4EmQsmml++xG96HGUi11MBTyNcMS EhlroxPRu7yKPo3nu9huPbDw8mL6F+/wD8T50DxCDE6m8rzNRpLbs73qvQL8qdcVxlkW ZyBhqum+Uf8GJEBnodx0ZkGjzCnSPA8Yiz2Cp04R899Ku6CsEhXyceGxAl3pQjEjCgiZ gbRInG0UVpq04A2TRWwzGqEb7zJtoLAbOYsG7uJZx7UoOrXybA4BY3xQWlMduOvGEd4G w88KZpG6bwd+ngWe8zf5uSuAYx7E5n1anYWPjywheeKfhsxY9YOMFwd/q5eyLFIbMBuB y4Ng== ARC-Authentication-Results: i=1; mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=Bg/dXvDS; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom=qemu-devel-bounces+patch=linaro.org@nongnu.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [2001:4830:134:3::11]) by mx.google.com with ESMTPS id s11-v6si1059042qvl.172.2018.05.25.07.34.03 for (version=TLS1 cipher=AES128-SHA bits=128/128); Fri, 25 May 2018 07:34:03 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) client-ip=2001:4830:134:3::11; Authentication-Results: mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=Bg/dXvDS; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom=qemu-devel-bounces+patch=linaro.org@nongnu.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:44367 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fMDn9-0003sl-7I for patch@linaro.org; Fri, 25 May 2018 10:34:03 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:45476) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fMDeJ-0005k7-Nc for qemu-devel@nongnu.org; Fri, 25 May 2018 10:24:57 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fMDeG-0003sr-Dr for qemu-devel@nongnu.org; Fri, 25 May 2018 10:24:55 -0400 Received: from mail-wr0-x243.google.com ([2a00:1450:400c:c0c::243]:37694) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fMDeG-0003sE-48 for qemu-devel@nongnu.org; Fri, 25 May 2018 10:24:52 -0400 Received: by mail-wr0-x243.google.com with SMTP id i12-v6so9592782wrc.4 for ; Fri, 25 May 2018 07:24:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Ilb+xV/1jTS1+/Rs3a4Ap9VqiRVoeMi1JNESVSUvGu8=; b=Bg/dXvDStFFtY6v8/7d+kYRzPfk/PlkmoCcF5lV9UojfaSsBQQPsXYd3+9c1rd9NkC DppmLfvKyuuOwt26M2Jpw1lXXjyfytUDcf1j4yjIXZbAy4f63Ga+q4Y3hrOf5aMXdkP+ rrfvbt0c4c9yy0WcVJruphI5z4uM3xAbWJtHE= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Ilb+xV/1jTS1+/Rs3a4Ap9VqiRVoeMi1JNESVSUvGu8=; b=RsUcmrRTHphFy+tH9F2VJkWdFaXl3qKtRcfOd3XIX9n9DVdrZUuaOEB5TePCaws6Oa D66QKCJjIuq82oNrfDFO87akWlHvhyxvYJks9PwE72X8thfempppDGvwrYhhbiMw7aD5 lSy1lrEXI5koOsaeyxNdU0cvVbPGFA7s0Ct2orZiJz6SZ7HE5Suh39Gh7QlP4AMW3Xvb BbR7yRg0+UQ4CJKR7NcRo78UErQBshsyGyho4SMmOUxQpyKTZffNEdqo9nALE3yC4JV+ XW1Fv3dv1vDTdt4NGtH9NDFeHBWqkMaTjPz4/THgRFjsmsroJnuz3KUyL2xjAyl10uL2 rgxg== X-Gm-Message-State: ALKqPwc+5FzOCUHxz3WgOm53BRlqFLmF1NlIMH4VNIjRwN+1ZlzL5rzG bNEFPVNW16lH9fwb6lnPgjF0/A== X-Received: by 2002:adf:deca:: with SMTP id i10-v6mr2545480wrn.37.1527258291003; Fri, 25 May 2018 07:24:51 -0700 (PDT) Received: from zen.linaro.local ([81.128.185.34]) by smtp.gmail.com with ESMTPSA id 123-v6sm12624041wmt.19.2018.05.25.07.24.48 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Fri, 25 May 2018 07:24:49 -0700 (PDT) Received: from zen.linaroharston (localhost [127.0.0.1]) by zen.linaro.local (Postfix) with ESMTP id 0843E3E025C; Fri, 25 May 2018 15:19:16 +0100 (BST) From: =?utf-8?q?Alex_Benn=C3=A9e?= To: cota@braap.org, famz@redhat.com, berrange@redhat.com, f4bug@amsat.org, richard.henderson@linaro.org, balrogg@gmail.com, aurelien@aurel32.net, agraf@suse.de Date: Fri, 25 May 2018 15:19:15 +0100 Message-Id: <20180525141915.6497-10-alex.bennee@linaro.org> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180525141915.6497-1-alex.bennee@linaro.org> References: <20180525133851.27161-1-alex.bennee@linaro.org> <20180525141915.6497-1-alex.bennee@linaro.org> MIME-Version: 1.0 X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2a00:1450:400c:c0c::243 Subject: [Qemu-devel] [PATCH v5 49/49] tests/tcg: add run, diff, and skip helper macros X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Peter Maydell , Eduardo Habkost , Yongbok Kim , qemu-devel@nongnu.org, "open list:ARM" , Paolo Bonzini , =?utf-8?q?Alex_Benn=C3=A9e?= , Richard Henderson Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" As we aren't using the default runners for all the test cases it is easy to miss out things like timeouts. To help with this we add some helpers and use them so we only need to make core changes in one place. Signed-off-by: Alex Bennée --- v5 - new for v5 --- tests/tcg/Makefile | 13 ++++++++++--- tests/tcg/aarch64/Makefile.target | 6 ++---- tests/tcg/arm/Makefile.target | 6 ++---- tests/tcg/i386/Makefile.target | 10 ++++------ tests/tcg/mips/Makefile.target | 3 +-- tests/tcg/multiarch/Makefile.target | 4 ++-- tests/tcg/sparc64/Makefile.target | 2 +- 7 files changed, 22 insertions(+), 22 deletions(-) -- 2.17.0 diff --git a/tests/tcg/Makefile b/tests/tcg/Makefile index 8158c1f099..ed33f3be8f 100644 --- a/tests/tcg/Makefile +++ b/tests/tcg/Makefile @@ -34,6 +34,15 @@ quiet-command = $(if $(V),$1,$(if $(2),@printf " %-7s %s\n" $2 $3 && $1, @$1)) +# $1 = test name, $2 = cmd, $3 = desc +run-test = $(call quiet-command, timeout $(TIMEOUT) $2 > $1.out,"TEST",$3) + +# $1 = test name, $2 = reference +diff-out = $(call quiet-command, diff -u $1.out $2 | head -n 10,"DIFF","$1.out with $2") + +# $1 = test name, $2 = reason +skip-test = @printf " SKIPPED %s because %s\n" $1 $2 + # Tests we are building TESTS= @@ -78,9 +87,7 @@ RUN_TESTS=$(patsubst %,run-%, $(TESTS)) RUN_TESTS+=$(EXTRA_RUNS) run-%: % - $(call quiet-command, \ - timeout $(TIMEOUT) $(QEMU) $< > $<.out, \ - "TEST", "$< on $(TARGET_NAME)") + $(call run-test, $<, $(QEMU) $<, "$< on $(TARGET_NAME)") .PHONY: run run: $(RUN_TESTS) diff --git a/tests/tcg/aarch64/Makefile.target b/tests/tcg/aarch64/Makefile.target index 7dba32138d..08c45b8470 100644 --- a/tests/tcg/aarch64/Makefile.target +++ b/tests/tcg/aarch64/Makefile.target @@ -13,7 +13,5 @@ TESTS:=$(AARCH64_TESTS) fcvt: LDFLAGS+=-lm run-fcvt: fcvt - $(call quiet-command, \ - $(QEMU) $< > fcvt.out && \ - diff -u $(AARCH64_SRC)/fcvt.ref fcvt.out, \ - "TEST", "$< (default) on $(TARGET_NAME)") + $(call run-test,$<,$(QEMU) $<, "$< on $(TARGET_NAME)") + $(call diff-out,$<,$(AARCH64_SRC)/fcvt.ref) diff --git a/tests/tcg/arm/Makefile.target b/tests/tcg/arm/Makefile.target index 14b5435fc6..aa4e4e3782 100644 --- a/tests/tcg/arm/Makefile.target +++ b/tests/tcg/arm/Makefile.target @@ -24,10 +24,8 @@ fcvt: LDFLAGS+=-lm # fcvt: CFLAGS+=-march=armv8.2-a+fp16 -mfpu=neon-fp-armv8 run-fcvt: fcvt - $(call quiet-command, \ - $(QEMU) $< > fcvt.out && \ - diff -u $(ARM_SRC)/fcvt.ref fcvt.out, \ - "TEST", "$< (default) on $(TARGET_NAME)") + $(call run-test,fcvt,$(QEMU) $<,"$< on $(TARGET_NAME)") + $(call diff-out,fcvt,$(ARM_SRC)/fcvt.ref) endif # On ARM Linux only supports 4k pages diff --git a/tests/tcg/i386/Makefile.target b/tests/tcg/i386/Makefile.target index d5c6f32b93..d482c57ad8 100644 --- a/tests/tcg/i386/Makefile.target +++ b/tests/tcg/i386/Makefile.target @@ -31,17 +31,15 @@ test-i386: test-i386.c test-i386-code16.S test-i386-vm86.S test-i386.h test-i386 # Specialist test runners run-runcom: runcom pi_10.com - $(call quiet-command, $(QEMU) ./runcom $(I386_SRC)/pi_10.com > runcom.out, "TEST", "$< on $(TARGET_NAME)") + $(call run-test,$<,$(QEMU) ./runcom $(I386_SRC)/pi_10.com,"$< on $(TARGET_NAME)") ifeq ($(SPEED), slow) run-test-i386-fprem: test-i386-fprem - $(call quiet-command, \ - $(QEMU) $< > $<.out && \ - diff -u $(I386_SRC)/$<.ref $<.out, \ - "TEST", "$< (default) on $(TARGET_NAME)") + $(call run-test,test-i386-fprem, $(QEMU) $<,"$< on $(TARGET_NAME)") + $(call diff-test,test-i386-fprem, $(I386_SRC)/$<.ref) else run-test-i386-fprem: test-i386-fprem - $(call quiet-command, /bin/true, "SLOW TEST", "$< SKIPPED on $(TARGET_NAME)") + $(call skip-test, $<, "SLOW") endif # On i386 and x86_64 Linux only supports 4k pages (large pages are a different hack) diff --git a/tests/tcg/mips/Makefile.target b/tests/tcg/mips/Makefile.target index 3db4eaf131..ae4780a5c7 100644 --- a/tests/tcg/mips/Makefile.target +++ b/tests/tcg/mips/Makefile.target @@ -19,5 +19,4 @@ hello-mips: LDFLAGS+=-nostdlib # however they are intended for system tests. run-hello-mips: hello-mips - $(call quiet-command, /bin/true, "BROKEN TEST", "$< SKIPPED on $(TARGET_NAME)") - + $(call skip-test,$<, "BROKEN") diff --git a/tests/tcg/multiarch/Makefile.target b/tests/tcg/multiarch/Makefile.target index 90e45a881e..b77084c146 100644 --- a/tests/tcg/multiarch/Makefile.target +++ b/tests/tcg/multiarch/Makefile.target @@ -27,10 +27,10 @@ testthread: LDFLAGS+=-lpthread # default case (host page size) run-test-mmap: test-mmap - $(call quiet-command, $(QEMU) $< > test-mmap.out, "TEST", \ + $(call run-test, test-mmap, $(QEMU) $<, \ "$< (default) on $(TARGET_NAME)") # additional page sizes (defined by each architecture adding to EXTRA_RUNS) run-test-mmap-%: test-mmap - $(call quiet-command, $(QEMU) -p $* $< > test-mmap-$*.out, "TEST", \ + $(call run-test, test-mmap-$*, $(QEMU) -p $* $<,\ "$< ($* byte pages) on $(TARGET_NAME)") diff --git a/tests/tcg/sparc64/Makefile.target b/tests/tcg/sparc64/Makefile.target index 7237779b26..064c7a598b 100644 --- a/tests/tcg/sparc64/Makefile.target +++ b/tests/tcg/sparc64/Makefile.target @@ -5,7 +5,7 @@ # different from the other hangs # /home/alex/lsrc/qemu/qemu.git/tests/tcg/multiarch/linux-test.c:264: Value too large for defined data type (ret=-1, errno=92/Value too large for defined data type) run-linux-test: linux-test - $(call quiet-command, /bin/true, "BROKEN TEST", "$< SKIPPED on $(TARGET_NAME)") + $(call skip-test, $<, "BROKEN") # On Sparc64 Linux support 8k pages EXTRA_RUNS+=run-test-mmap-8192