From patchwork Fri May 25 15:05:53 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Alex_Benn=C3=A9e?= X-Patchwork-Id: 136938 Delivered-To: patch@linaro.org Received: by 2002:a2e:9706:0:0:0:0:0 with SMTP id r6-v6csp3707020lji; Fri, 25 May 2018 08:06:26 -0700 (PDT) X-Google-Smtp-Source: ADUXVKK5NmnNCai09U00QgVuEnXSAuzi+sqeWMvQpgMbb3cm+X75cSmAW4SQW3mN0qRorfUngOsn X-Received: by 2002:a0c:e410:: with SMTP id o16-v6mr2575543qvl.66.1527260786853; Fri, 25 May 2018 08:06:26 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1527260786; cv=none; d=google.com; s=arc-20160816; b=QCNWPv2By1rjiDlFB8ZKDSIkLQv14k08wxiTTsHk8nRt9358P8v+/cAxMyz+gWFF6+ lgOHNZkYyqOcfaLQV7wIPwfanOM3qeR6UGlewfbCHoF3r7l2kFqxHFOrgC31giWh4fPd u5btfUBt++eRGj4ZAsF3dBIrxnK95U10h3gNs6Qu9hwdkDN5TWpYk5+sIZFGKkjCUcWK 2NKEBel52GqX0+PPrYel9Py5Dx6k57cOMo7mF3BIDvKQ/NgzWXy9G06/DKOTva6w1wMj ZFWaD9WATUEHCOMz0ZRBMZbpq3Uu+6MkU41Be08So3K/s2TNhEnU+Dhc/uX/eoOf1CGB fN7Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:cc:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject :content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:arc-authentication-results; bh=gzKC/GpL5RjnMsLLFUhVe1yFn3VvwnUoaqZ183ctof4=; b=nLo9egjruUFYcoyBvYJbwABjBjUO9XyEfwrAHuLnlXwCkTyb0prkgCC6BBxkAH/1OA Ou/ow15YS8/YQqjCnZRP+uub925mzOE4rp74GCSggYUfpYjO/scUoFhP0IHDenFbnqQ0 4OHonc3nP/95OtoIuz1gLnzOcx+cNxnxX6xkz0moFXq/R0DLEaMyg9FqStLeNn7Dt1pj JhkQbDCfZ9bkE9zktzTYyB4muxYKGSE0293n3jbCqCijdUgji/oiyXykAlslaoCCaN2P nNcNVDOtYZgV9DbRsHBXjAsRQGYG1ugseZsE2ILPkNp3flz+HCVYANPsRLxTmOsmIp0W QDIQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=Ty9FdMnB; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom=qemu-devel-bounces+patch=linaro.org@nongnu.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [2001:4830:134:3::11]) by mx.google.com with ESMTPS id y56-v6si4297794qty.198.2018.05.25.08.06.26 for (version=TLS1 cipher=AES128-SHA bits=128/128); Fri, 25 May 2018 08:06:26 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) client-ip=2001:4830:134:3::11; Authentication-Results: mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=Ty9FdMnB; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom=qemu-devel-bounces+patch=linaro.org@nongnu.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:44536 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fMEIU-0004UV-AD for patch@linaro.org; Fri, 25 May 2018 11:06:26 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:53975) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fMEI6-0004Rl-JU for qemu-devel@nongnu.org; Fri, 25 May 2018 11:06:04 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fMEI0-0003ot-IB for qemu-devel@nongnu.org; Fri, 25 May 2018 11:06:02 -0400 Received: from mail-wm0-x241.google.com ([2a00:1450:400c:c09::241]:50808) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fMEI0-0003nP-7u for qemu-devel@nongnu.org; Fri, 25 May 2018 11:05:56 -0400 Received: by mail-wm0-x241.google.com with SMTP id t11-v6so15314692wmt.0 for ; Fri, 25 May 2018 08:05:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=gzKC/GpL5RjnMsLLFUhVe1yFn3VvwnUoaqZ183ctof4=; b=Ty9FdMnBbqXanwecBMyYymK06p4AKNIDJkTVbzcVv6lZ6jnHcZnIilNcUD1z0vddNV dS4r5NRUXFq2lGvY4wEgQj5Va/fjk/B/IdvJJrNnkjpcPuL8IlcNZ1/gjmamifBBJFcq aZ4plDN99BmO95LhEyHA7idhF1cXEdbseSHYc= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=gzKC/GpL5RjnMsLLFUhVe1yFn3VvwnUoaqZ183ctof4=; b=GDUx36G9E9RfpUYWHIvtl8nj6yqZ2/+d052mwueGbODOOzTVw17DgMP+TER54n5You QByEG9nfs0X+rZTNOo6/9BskPwInJVtKywF1H5yDEnHUqTB37GfwzAbtW6nbW/m7qmLB e8y/t7NI1XKWMaxHFo0ezdCaVasu3EzaQoM1aRQk6055YXAIkTEPiEvz4fvn+mkVx9xs uP56BLqRLbi5A+sQAULApZzOTajorQOKhhoDa/q77p38w/9569YgLwm+l3sMRkUPOxhz H9oHU/r1du6CRV7IpVtiFmqyOwaLvx8jHr3qmL7LtwJie8nWTFIljM1bWA3HOx4WRg2S skgA== X-Gm-Message-State: ALKqPwdAMmK06VLe88YTczpgdSwEj11PqsLG3uY8vhekCLwt3Q34U0Es 2PnSxHYT3QDNoO2aeNtb9FJlKw== X-Received: by 2002:a1c:d046:: with SMTP id h67-v6mr2010963wmg.80.1527260755084; Fri, 25 May 2018 08:05:55 -0700 (PDT) Received: from zen.linaro.local ([81.128.185.34]) by smtp.gmail.com with ESMTPSA id o53-v6sm35543347wrc.96.2018.05.25.08.05.54 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Fri, 25 May 2018 08:05:54 -0700 (PDT) Received: from zen.linaroharston (localhost [127.0.0.1]) by zen.linaro.local (Postfix) with ESMTP id E46B73E025C; Fri, 25 May 2018 16:05:53 +0100 (BST) From: =?utf-8?q?Alex_Benn=C3=A9e?= To: cota@braap.org, famz@redhat.com, berrange@redhat.com, f4bug@amsat.org, richard.henderson@linaro.org, balrogg@gmail.com, aurelien@aurel32.net, agraf@suse.de Date: Fri, 25 May 2018 16:05:53 +0100 Message-Id: <20180525150553.26741-1-alex.bennee@linaro.org> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180525141915.6497-10-alex.bennee@linaro.org> References: <20180525141915.6497-10-alex.bennee@linaro.org> MIME-Version: 1.0 X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2a00:1450:400c:c09::241 Subject: [Qemu-devel] [PATCH v5.1 49/49] tests/tcg: add run, diff, and skip helper macros X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Peter Maydell , Eduardo Habkost , Yongbok Kim , qemu-devel@nongnu.org, "open list:ARM" , Paolo Bonzini , =?utf-8?q?Alex_Benn=C3=A9e?= , Richard Henderson Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" As we aren't using the default runners for all the test cases it is easy to miss out things like timeouts. To help with this we add some helpers and use them so we only need to make core changes in one place. Signed-off-by: Alex Bennée --- v5 - new for v5 v5.1 - diff-test->diff-out --- tests/tcg/Makefile | 13 ++++++++++--- tests/tcg/aarch64/Makefile.target | 6 ++---- tests/tcg/arm/Makefile.target | 6 ++---- tests/tcg/i386/Makefile.target | 10 ++++------ tests/tcg/mips/Makefile.target | 3 +-- tests/tcg/multiarch/Makefile.target | 4 ++-- tests/tcg/sparc64/Makefile.target | 2 +- 7 files changed, 22 insertions(+), 22 deletions(-) -- 2.17.0 diff --git a/tests/tcg/Makefile b/tests/tcg/Makefile index 8158c1f099..ed33f3be8f 100644 --- a/tests/tcg/Makefile +++ b/tests/tcg/Makefile @@ -34,6 +34,15 @@ quiet-command = $(if $(V),$1,$(if $(2),@printf " %-7s %s\n" $2 $3 && $1, @$1)) +# $1 = test name, $2 = cmd, $3 = desc +run-test = $(call quiet-command, timeout $(TIMEOUT) $2 > $1.out,"TEST",$3) + +# $1 = test name, $2 = reference +diff-out = $(call quiet-command, diff -u $1.out $2 | head -n 10,"DIFF","$1.out with $2") + +# $1 = test name, $2 = reason +skip-test = @printf " SKIPPED %s because %s\n" $1 $2 + # Tests we are building TESTS= @@ -78,9 +87,7 @@ RUN_TESTS=$(patsubst %,run-%, $(TESTS)) RUN_TESTS+=$(EXTRA_RUNS) run-%: % - $(call quiet-command, \ - timeout $(TIMEOUT) $(QEMU) $< > $<.out, \ - "TEST", "$< on $(TARGET_NAME)") + $(call run-test, $<, $(QEMU) $<, "$< on $(TARGET_NAME)") .PHONY: run run: $(RUN_TESTS) diff --git a/tests/tcg/aarch64/Makefile.target b/tests/tcg/aarch64/Makefile.target index 7dba32138d..08c45b8470 100644 --- a/tests/tcg/aarch64/Makefile.target +++ b/tests/tcg/aarch64/Makefile.target @@ -13,7 +13,5 @@ TESTS:=$(AARCH64_TESTS) fcvt: LDFLAGS+=-lm run-fcvt: fcvt - $(call quiet-command, \ - $(QEMU) $< > fcvt.out && \ - diff -u $(AARCH64_SRC)/fcvt.ref fcvt.out, \ - "TEST", "$< (default) on $(TARGET_NAME)") + $(call run-test,$<,$(QEMU) $<, "$< on $(TARGET_NAME)") + $(call diff-out,$<,$(AARCH64_SRC)/fcvt.ref) diff --git a/tests/tcg/arm/Makefile.target b/tests/tcg/arm/Makefile.target index 14b5435fc6..aa4e4e3782 100644 --- a/tests/tcg/arm/Makefile.target +++ b/tests/tcg/arm/Makefile.target @@ -24,10 +24,8 @@ fcvt: LDFLAGS+=-lm # fcvt: CFLAGS+=-march=armv8.2-a+fp16 -mfpu=neon-fp-armv8 run-fcvt: fcvt - $(call quiet-command, \ - $(QEMU) $< > fcvt.out && \ - diff -u $(ARM_SRC)/fcvt.ref fcvt.out, \ - "TEST", "$< (default) on $(TARGET_NAME)") + $(call run-test,fcvt,$(QEMU) $<,"$< on $(TARGET_NAME)") + $(call diff-out,fcvt,$(ARM_SRC)/fcvt.ref) endif # On ARM Linux only supports 4k pages diff --git a/tests/tcg/i386/Makefile.target b/tests/tcg/i386/Makefile.target index cd173363ee..97b7c23cf1 100644 --- a/tests/tcg/i386/Makefile.target +++ b/tests/tcg/i386/Makefile.target @@ -31,7 +31,7 @@ test-i386: test-i386.c test-i386-code16.S test-i386-vm86.S test-i386.h test-i386 # Specialist test runners run-runcom: runcom pi_10.com - $(call quiet-command, $(QEMU) ./runcom $(I386_SRC)/pi_10.com > runcom.out, "TEST", "$< on $(TARGET_NAME)") + $(call run-test,$<,$(QEMU) ./runcom $(I386_SRC)/pi_10.com,"$< on $(TARGET_NAME)") ifeq ($(SPEED), slow) @@ -40,13 +40,11 @@ test-i386-fprem.ref: test-i386-fprem run-test-i386-fprem: TIMEOUT=60 run-test-i386-fprem: test-i386-fprem - $(call quiet-command, \ - $(QEMU) $< > $<.out && \ - diff -u $(I386_SRC)/$<.ref $<.out, \ - "TEST", "$< (default) on $(TARGET_NAME)") + $(call run-test,test-i386-fprem, $(QEMU) $<,"$< on $(TARGET_NAME)") + $(call diff-out,test-i386-fprem, $(I386_SRC)/$<.ref) else run-test-i386-fprem: test-i386-fprem - $(call quiet-command, /bin/true, "SLOW TEST", "$< SKIPPED on $(TARGET_NAME)") + $(call skip-test, $<, "SLOW") endif # On i386 and x86_64 Linux only supports 4k pages (large pages are a different hack) diff --git a/tests/tcg/mips/Makefile.target b/tests/tcg/mips/Makefile.target index 3db4eaf131..ae4780a5c7 100644 --- a/tests/tcg/mips/Makefile.target +++ b/tests/tcg/mips/Makefile.target @@ -19,5 +19,4 @@ hello-mips: LDFLAGS+=-nostdlib # however they are intended for system tests. run-hello-mips: hello-mips - $(call quiet-command, /bin/true, "BROKEN TEST", "$< SKIPPED on $(TARGET_NAME)") - + $(call skip-test,$<, "BROKEN") diff --git a/tests/tcg/multiarch/Makefile.target b/tests/tcg/multiarch/Makefile.target index 90e45a881e..b77084c146 100644 --- a/tests/tcg/multiarch/Makefile.target +++ b/tests/tcg/multiarch/Makefile.target @@ -27,10 +27,10 @@ testthread: LDFLAGS+=-lpthread # default case (host page size) run-test-mmap: test-mmap - $(call quiet-command, $(QEMU) $< > test-mmap.out, "TEST", \ + $(call run-test, test-mmap, $(QEMU) $<, \ "$< (default) on $(TARGET_NAME)") # additional page sizes (defined by each architecture adding to EXTRA_RUNS) run-test-mmap-%: test-mmap - $(call quiet-command, $(QEMU) -p $* $< > test-mmap-$*.out, "TEST", \ + $(call run-test, test-mmap-$*, $(QEMU) -p $* $<,\ "$< ($* byte pages) on $(TARGET_NAME)") diff --git a/tests/tcg/sparc64/Makefile.target b/tests/tcg/sparc64/Makefile.target index 7237779b26..064c7a598b 100644 --- a/tests/tcg/sparc64/Makefile.target +++ b/tests/tcg/sparc64/Makefile.target @@ -5,7 +5,7 @@ # different from the other hangs # /home/alex/lsrc/qemu/qemu.git/tests/tcg/multiarch/linux-test.c:264: Value too large for defined data type (ret=-1, errno=92/Value too large for defined data type) run-linux-test: linux-test - $(call quiet-command, /bin/true, "BROKEN TEST", "$< SKIPPED on $(TARGET_NAME)") + $(call skip-test, $<, "BROKEN") # On Sparc64 Linux support 8k pages EXTRA_RUNS+=run-test-mmap-8192