From patchwork Tue Jun 26 14:35:38 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Alex_Benn=C3=A9e?= X-Patchwork-Id: 139989 Delivered-To: patch@linaro.org Received: by 2002:a2e:970d:0:0:0:0:0 with SMTP id r13-v6csp5323188lji; Tue, 26 Jun 2018 07:36:54 -0700 (PDT) X-Google-Smtp-Source: AAOMgpdGTwzMhV1+QuD6DQIoeZ98Znx3ofKVVCcJLzoAbgH5XMPkvIBGjezSuyZThirgL/vy6pmV X-Received: by 2002:ac8:34f:: with SMTP id w15-v6mr1606716qtg.410.1530023814232; Tue, 26 Jun 2018 07:36:54 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1530023814; cv=none; d=google.com; s=arc-20160816; b=SxzajS3USv9tuLVIvcijj8KjAfSgB17B5QUAAVvDHP5OC9rqvt7RsXAxQDgepoxQjz QAkN3XQHxkFlbvmrTuIM7GONhkmeUfHJrcwF2sIiBbWEtp7/agwcCQ+y8cJyztlDCnCH GvIs2lcipOaHjtHbaNROBz1jyHXeIKCR02s8xmE/v8xbeOmmADuCOiXmm4q4bt0IFysv FmKcRhBXp5Pi7Sw0LOBGFgSpHyWXG8PLqDI7XHZR/8cpd3J0DuDvBtC/7JwT3D/lB+a3 Tawp9A2Ec8VEd2Hb6LTKvgYTs/MI2tGPAZgoBNYZ+lH9RIz8Jk8siLjZ8DtAIF+6oEIs g1uA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:cc:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject :content-transfer-encoding:mime-version:message-id:date:to:from :dkim-signature:arc-authentication-results; bh=hs2wOBTm+KB0y6OTQlHrynkBztfYpfwyGvLd0ouTB18=; b=XzVj4J98Xmw+/qsXBLXHfIdweL6tEX3Jse+4RDF61PNX+2Z8cRIT0gCOcS3fA5qlai g/VIrL1J30DjYXOdeI4X4A9e5mecwMAOJ764sl1ZfOWhUUFDlZHQOdXC1DVSWWqDn6kS rkia+0FUeYFZVB1q8fukrxwLUjOIxUNsg7uHw1TV9zI8Ef6QtwM0mKpo0CMENeCvKHl8 LoR9pU3uCJncUVRFNEJXWoEDGQPG93AoEUTDHT2bMb3heFuWKKh0LDMy8bu7wRyeqnMA 2F5LWSbKS61SIISdv9i08qjMm9+/CkX3VGqdVGYTInsKeOPLBvYxiv3uOe2HAiIYO+Mh MDug== ARC-Authentication-Results: i=1; mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=VJV+awDU; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [2001:4830:134:3::11]) by mx.google.com with ESMTPS id a1-v6si249640qvc.52.2018.06.26.07.36.54 for (version=TLS1 cipher=AES128-SHA bits=128/128); Tue, 26 Jun 2018 07:36:54 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) client-ip=2001:4830:134:3::11; Authentication-Results: mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=VJV+awDU; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:53095 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fXp5R-0001Hl-Iy for patch@linaro.org; Tue, 26 Jun 2018 10:36:53 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:45610) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fXp4N-0000qN-55 for qemu-devel@nongnu.org; Tue, 26 Jun 2018 10:35:48 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fXp4J-0005nG-94 for qemu-devel@nongnu.org; Tue, 26 Jun 2018 10:35:47 -0400 Received: from mail-wr0-x235.google.com ([2a00:1450:400c:c0c::235]:39968) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fXp4I-0005mm-Kp for qemu-devel@nongnu.org; Tue, 26 Jun 2018 10:35:43 -0400 Received: by mail-wr0-x235.google.com with SMTP id g18-v6so17510795wro.7 for ; Tue, 26 Jun 2018 07:35:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=hs2wOBTm+KB0y6OTQlHrynkBztfYpfwyGvLd0ouTB18=; b=VJV+awDUmEVgtSnGwhQh64Oe2XT2K1szMaLlRB7b0ZZzIKSMo4R5bnWoAoF/fgGNRI hqTnh3T2mZdZ2AfOM0Nd6Cj/hgPoF0oSTgXq+7kc5nFqEWISiLAuVmcYlMd0Y5IBNcxl dI+MivUx4sGkRQH/JaNr/B8kr9TOWmH1pB9lM= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=hs2wOBTm+KB0y6OTQlHrynkBztfYpfwyGvLd0ouTB18=; b=bC8xvJhC3jQ03/biDqynpv0iiqtFxJSTuqhDuMYmmgwdQuKhhacz+ljjvs0b8IcH7m BGQpv0NcmrG/ula2NspA/IzJqGGrfK9/RnlcvXSvXpVNmk+LzdkKjkhPQnGqSZcPT5tD D5BRP3gO3KacNMI9c8h9QuwSxdPqEpml5gOcjMCLg7XJprZ22laSqzXhbFRqgYgMn0Vy +a0gqAJW4VTh/9dZlGu4XDzPvZXFJfXGnkMsBrQ1VfTL8TuXp33OhsTTcLSpsz0vSSBc cYugFtQWVU2n5m6rsp5KUdoBRBki4M7OE0vi2a3gfvOS/q3XuNE3WVpZJQ2Q7Ezz62w5 /oGg== X-Gm-Message-State: APt69E2M05tkg7Op5+y0PJIqLeX8aTEv1YRxI+HIwyhPUQjyQtrfVtJo 07BUFldvAVo+wDlMzo6i1G+jrQ== X-Received: by 2002:adf:a581:: with SMTP id g1-v6mr1792365wrc.233.1530023741334; Tue, 26 Jun 2018 07:35:41 -0700 (PDT) Received: from zen.linaro.local ([81.128.185.34]) by smtp.gmail.com with ESMTPSA id o17-v6sm1995601wrp.62.2018.06.26.07.35.40 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 26 Jun 2018 07:35:40 -0700 (PDT) Received: from zen.linaroharston (localhost [127.0.0.1]) by zen.linaro.local (Postfix) with ESMTP id 191453E01DB; Tue, 26 Jun 2018 15:35:40 +0100 (BST) From: =?utf-8?q?Alex_Benn=C3=A9e?= To: qemu-devel@nongnu.org Date: Tue, 26 Jun 2018 15:35:38 +0100 Message-Id: <20180626143538.5390-1-alex.bennee@linaro.org> X-Mailer: git-send-email 2.17.1 MIME-Version: 1.0 X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2a00:1450:400c:c0c::235 Subject: [Qemu-devel] [PATCH] blogposts: add post about the new check-tcg infrastructure X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: pbonzini@redhat.com, =?utf-8?q?Alex_Benn=C3=A9e?= Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" Signed-off-by: Alex Bennée --- v2 - changes to the language for the lesser known architectures --- _posts/2018-06-21-tcg-testing.md | 139 +++++++++++++++++++++++++++++++ 1 file changed, 139 insertions(+) create mode 100644 _posts/2018-06-21-tcg-testing.md -- 2.17.1 diff --git a/_posts/2018-06-21-tcg-testing.md b/_posts/2018-06-21-tcg-testing.md new file mode 100644 index 0000000..dfc504c --- /dev/null +++ b/_posts/2018-06-21-tcg-testing.md @@ -0,0 +1,139 @@ +--- +layout: post +title: "QEMU TCG Tests" +date: 2018-06-21 10:30:00:00 +0000 +last_modified_at: 2018-06-21 10:30:00:00 +0000 +author: Alex Bennée +categories: [testing, docker, compilation, tcg] +--- + +Ever since I started working on QEMU there was a small directory +called tests/tcg that was in a perpetually broken state. It contains +tests that exercise QEMU's ability to work across architectures using +the power of the Tiny Code Generator. However as these tests needed to +be compiled for the *guest* architectures and not the *host* +architecture - known as cross-compiling - most developers never ran +them. As the tests were hardly ever built inevitably a certain amount +of bit-rot set in. + +# Cross Compilers + +In the old days cross-compilation setups were almost all hand-crafted +affairs which involved building versions of binutils, gcc and a basic +libc. If you couldn't get someone to give you a pre-built tarball it +was something you laboured through once and hopefully never had to +touch again. There were even dedicated scripts like crosstool-ng which +attempted to make the process of patching and configuring your +toolchain easier. + +While the distributions have improved their support for cross +compilers over the years there are still plenty of variations in how +they are deployed. It is hard for a project like QEMU which has to +build on a wide range of operating systems and architectures to +seamlessly use any given distributions compiler setup. However for +those with cross compilers to hand `configure` now accepts two +additional flags: + + --cross-cc-$(ARCH) + --cross-cc-flags-$(ARCH) + +With a compiler specified for each guest architecture you want to test +the build system can now build and run the tests. However for +developers that don't have cross compilers around they can now take +advantage of QEMU's docker images. + +# Enter Docker Containers + +If you work in IT you would be hard pressed not to have noticed the +hype around Docker and the concept of containerisation over the last +few years. Put simply containers allow you to define a known working +set of software that gets run in an isolated environment for a given +task. While this has many uses for QEMU it allows us to define build +environments that any developer can run without having to mess around +with their preferred host setup. + +Over the last few years QEMU's build system has been expanding the +number of docker images it supports. Most of this has been in service +of our CI testing such as [Patchew](https://patchew.org/QEMU/) and +[Shippable](https://app.shippable.com/github/qemu/qemu/dashboard) but +any developer with a docker setup can run the exact same images. For +example if you want to check your patches won't break when compiled on +a 32 bit ARM system you can run: + + make docker-test-build@debian-armhf-cross J=n + +instead of tracking down a piece of ARM hardware to actually build on. +Run `make docker` in your source tree to see the range of builds and +tests it can support. + +# make check-tcg + +With the latest work [merged into +master](https://git.qemu.org/?p=qemu.git;a=commit;h=de44c044420d1139480fa50c2d5be19223391218) we can now +take advantage of both hand configured and docker based cross +compilers to build test cases for TCG again. To run the TCG tests +after you have built QEMU: + + make check-tcg + +and the build system will build and run all the tests it can for your +configured targets. + +# Rules for tests/tcg + +So now we have the infrastructure in place to add new tests what rules +need to be followed to add new tests? + +Well the first thing to note is currently all the tests are for +linux-user versions of QEMU. This means the tests are all currently +user-space tests that have access to the Linux syscall ABI. + +Another thing to note is the tests are standalone from the rest of the +QEMU test infrastructure. To keep things simple they are compiled as +standalone "static" binaries. As the cross-compilation setup can be +quite rudimentary for some of the rarer architectures we only compile +against a standard libc. There is no support for linking to other +libraries like for example glib. Thread and maths support is part of +glibc so shouldn't be a problem. + +Finally when writing new tests consider if it really is architecture +specific or can be added to `tests/tcg/multiarch`. The multiarch tests +are re-built for every supported architecture and should be the +default place for anything explicitly testing syscalls and other +common parts of the code base. + +# What's Next + +My hope with this work is we can start adding more tests to +systematically defend functionality in linux-user. In fact I hope the +first port of call to reproducing a crash would be writing a test case +that can be added to our growing library of tests. + +Another thing that needs sorting out is getting toolchains for all of +the less common architectures. The current work relies heavily on the +excellent work of the Debian toolchain team in making multiarch +aware cross compilers available in their distribution. However QEMU +supports a lot more architectures than QEMU, some only as system +emulations. In principle supporting them is as easy as adding another +docker recipe but it might be these recipes end up having to compile +the compilers from source. + +The `tests/tcg` directory still contains a number of source files we +don't build. + +The cris and openrisc directories contain user-space tests which just +need the support of a toolchain and the relevant Makefile plumbing to +be added. + +The lm32, mips and xtensa targets have a set of tests that need a +system emulator. Aside from adding the compilers as docker images some +additional work is needed to handle the differences between plain +linux-user tests which can simply return an exit code to getting the +results from a qemu-system emulation. Some architectures have +semi-hosting support already for this while others report their test +status over a simple serial link which will need to be parsed and +handled in custom versions of the +[`run-%:`](https://git.qemu.org/?p=qemu.git;a=blob;f=tests/tcg/Makefile;h=bf064153900a438e4ad8e2d79eaaac8a27d66062;hb=HEAD#l95) +rule. + +