diff mbox series

blogposts: add post about the new check-tcg infrastructure

Message ID 20180626143538.5390-1-alex.bennee@linaro.org
State New
Headers show
Series blogposts: add post about the new check-tcg infrastructure | expand

Commit Message

Alex Bennée June 26, 2018, 2:35 p.m. UTC 
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>


---
v2
  - changes to the language for the lesser known architectures
---
 _posts/2018-06-21-tcg-testing.md | 139 +++++++++++++++++++++++++++++++
 1 file changed, 139 insertions(+)
 create mode 100644 _posts/2018-06-21-tcg-testing.md

-- 
2.17.1

Comments

Paolo Bonzini June 28, 2018, 10:45 a.m. UTC | #1 
On 26/06/2018 16:35, Alex Bennée wrote:
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>


Pushed with a little bit of editing, thanks!

diff --git a/_posts/2018-06-21-tcg-testing.md b/_posts/2018-06-28-tcg-testing.md
similarity index 83%
rename from _posts/2018-06-21-tcg-testing.md
rename to _posts/2018-06-28-tcg-testing.md
index e61312e..026e726 100644
--- a/_posts/2018-06-21-tcg-testing.md
+++ b/_posts/2018-06-28-tcg-testing.md
@@ -7,39 +7,39 @@ author: Alex Bennée
 categories: [testing, docker, compilation, tcg]
 ---
 
-Ever since I started working on QEMU there was a small directory
-called tests/tcg that was in a perpetually broken state. It contains
+Ever since I started working on QEMU, a small directory
+called `tests/tcg` has been in a perpetually broken state. It contains
 tests that exercise QEMU's ability to work across architectures using
 the power of the Tiny Code Generator. However as these tests needed to
 be compiled for the *guest* architectures and not the *host*
-architecture - known as cross-compiling - most developers never ran
-them. As the tests were hardly ever built inevitably a certain amount
-of bit-rot set in.
+architecture&mdash;this is known as cross-compiling&mdash;most developers
+never ran them. As the tests were hardly ever built inevitably a certain
+amount of bit-rot set in.
 
 # Cross Compilers
 
-In the old days cross-compilation setups were almost all hand-crafted
+In the old days, cross-compilation setups were almost all hand-crafted
 affairs which involved building versions of binutils, gcc and a basic
-libc. If you couldn't get someone to give you a pre-built tarball it
+libc. If you couldn't get someone to give you a pre-built tarball, it
 was something you laboured through once and hopefully never had to
 touch again. There were even dedicated scripts like crosstool-ng which
 attempted to make the process of patching and configuring your
 toolchain easier.
 
 While the distributions have improved their support for cross
-compilers over the years there are still plenty of variations in how
+compilers over the years, there are still plenty of variations in how
 they are deployed. It is hard for a project like QEMU which has to
 build on a wide range of operating systems and architectures to
 seamlessly use any given distributions compiler setup. However for
-those with cross compilers to hand `configure` now accepts two
+those with cross compilers at hand `configure` now accepts two
 additional flags:
 
     --cross-cc-$(ARCH)
     --cross-cc-flags-$(ARCH)
 
 With a compiler specified for each guest architecture you want to test
-the build system can now build and run the tests. However for
-developers that don't have cross compilers around they can now take
+the build system can now build and run the tests. For
+developers that don't have cross compilers around, they can take
 advantage of QEMU's docker images.
 
 # Enter Docker Containers
@@ -66,11 +66,11 @@ instead of tracking down a piece of ARM hardware to actually build on.
 Run `make docker` in your source tree to see the range of builds and
 tests it can support.
 
-# make check-tcg
+# `make check-tcg`
 
 With the latest work [merged into
 master](https://git.qemu.org/?p=qemu.git;a=commit;h=de44c044420d1139480fa50c2d5be19223391218) we can now
-take advantage of both hand configured and docker based cross
+take advantage of either hand-configured and Docker-based cross
 compilers to build test cases for TCG again. To run the TCG tests
 after you have built QEMU:
 
@@ -79,16 +79,16 @@ after you have built QEMU:
 and the build system will build and run all the tests it can for your
 configured targets.
 
-# Rules for tests/tcg
+# Rules for `tests/tcg`
 
 So now we have the infrastructure in place to add new tests what rules
 need to be followed to add new tests? 
 
-Well the first thing to note is currently all the tests are for
-linux-user versions of QEMU. This means the tests are all currently
+Well the first thing to note is currently all the tests are for the
+`linux-user` variant of QEMU. This means the tests are all currently
 user-space tests that have access to the Linux syscall ABI.
 
-Another thing to note is the tests are standalone from the rest of the
+Another thing to note is the tests are separate from the rest of the
 QEMU test infrastructure. To keep things simple they are compiled as
 standalone "static" binaries. As the cross-compilation setup can be
 quite rudimentary for some of the rarer architectures we only compile
@@ -99,10 +99,10 @@ glibc so shouldn't be a problem.
 Finally when writing new tests consider if it really is architecture
 specific or can be added to `tests/tcg/multiarch`. The multiarch tests
 are re-built for every supported architecture and should be the
-default place for anything explicitly testing syscalls and other
+default place for anything that tests syscalls or other
 common parts of the code base.
 
-# What's Next
+# What's next
 
 My hope with this work is we can start adding more tests to
 systematically defend functionality in linux-user. In fact I hope the
@@ -135,5 +135,3 @@ status over a simple serial link which will need to be parsed and
 handled in custom versions of the
 [`run-%:`](https://git.qemu.org/?p=qemu.git;a=blob;f=tests/tcg/Makefile;h=bf064153900a438e4ad8e2d79eaaac8a27d66062;hb=HEAD#l95)
 rule.
-
-
no-reply@patchew.org July 2, 2018, 2:21 a.m. UTC | #2 
Hi,

This series seems to have some coding style problems. See output below for
more information:

Type: series
Message-id: 20180626143538.5390-1-alex.bennee@linaro.org
Subject: [Qemu-devel] [PATCH] blogposts: add post about the new check-tcg infrastructure

=== TEST SCRIPT BEGIN ===
#!/bin/bash

BASE=base
n=1
total=$(git log --oneline $BASE.. | wc -l)
failed=0

git config --local diff.renamelimit 0
git config --local diff.renames True
git config --local diff.algorithm histogram

commits="$(git log --format=%H --reverse $BASE..)"
for c in $commits; do
    echo "Checking PATCH $n/$total: $(git log -n 1 --format=%s $c)..."
    if ! git show $c --format=email | ./scripts/checkpatch.pl --mailback -; then
        failed=1
        echo
    fi
    n=$((n+1))
done

exit $failed
=== TEST SCRIPT END ===

Updating 3c8cf5a9c21ff8782164d1def7f44bd888713384
Switched to a new branch 'test'
1efad9bff0 blogposts: add post about the new check-tcg infrastructure

=== OUTPUT BEGIN ===
Checking PATCH 1/1: blogposts: add post about the new check-tcg infrastructure...
WARNING: added, moved or deleted file(s), does MAINTAINERS need updating?
#14: 
new file mode 100644

ERROR: trailing whitespace
#103: FILE: _posts/2018-06-21-tcg-testing.md:85:
+need to be followed to add new tests? $

ERROR: trailing whitespace
#140: FILE: _posts/2018-06-21-tcg-testing.md:122:
+don't build. $

total: 2 errors, 1 warnings, 139 lines checked

Your patch has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.

=== OUTPUT END ===

Test command exited with code: 1


---
Email generated automatically by Patchew [http://patchew.org/].
Please send your feedback to patchew-devel@redhat.com
diff mbox series

Patch

diff --git a/_posts/2018-06-21-tcg-testing.md b/_posts/2018-06-21-tcg-testing.md
new file mode 100644
index 0000000..dfc504c
--- /dev/null
+++ b/_posts/2018-06-21-tcg-testing.md
@@ -0,0 +1,139 @@ 
+---
+layout: post
+title:  "QEMU TCG Tests"
+date:   2018-06-21 10:30:00:00 +0000
+last_modified_at: 2018-06-21 10:30:00:00 +0000
+author: Alex Bennée
+categories: [testing, docker, compilation, tcg]
+---
+
+Ever since I started working on QEMU there was a small directory
+called tests/tcg that was in a perpetually broken state. It contains
+tests that exercise QEMU's ability to work across architectures using
+the power of the Tiny Code Generator. However as these tests needed to
+be compiled for the *guest* architectures and not the *host*
+architecture - known as cross-compiling - most developers never ran
+them. As the tests were hardly ever built inevitably a certain amount
+of bit-rot set in.
+
+# Cross Compilers
+
+In the old days cross-compilation setups were almost all hand-crafted
+affairs which involved building versions of binutils, gcc and a basic
+libc. If you couldn't get someone to give you a pre-built tarball it
+was something you laboured through once and hopefully never had to
+touch again. There were even dedicated scripts like crosstool-ng which
+attempted to make the process of patching and configuring your
+toolchain easier.
+
+While the distributions have improved their support for cross
+compilers over the years there are still plenty of variations in how
+they are deployed. It is hard for a project like QEMU which has to
+build on a wide range of operating systems and architectures to
+seamlessly use any given distributions compiler setup. However for
+those with cross compilers to hand `configure` now accepts two
+additional flags:
+
+    --cross-cc-$(ARCH)
+    --cross-cc-flags-$(ARCH)
+
+With a compiler specified for each guest architecture you want to test
+the build system can now build and run the tests. However for
+developers that don't have cross compilers around they can now take
+advantage of QEMU's docker images.
+
+# Enter Docker Containers
+
+If you work in IT you would be hard pressed not to have noticed the
+hype around Docker and the concept of containerisation over the last
+few years. Put simply containers allow you to define a known working
+set of software that gets run in an isolated environment for a given
+task. While this has many uses for QEMU it allows us to define build
+environments that any developer can run without having to mess around
+with their preferred host setup.
+
+Over the last few years QEMU's build system has been expanding the
+number of docker images it supports. Most of this has been in service
+of our CI testing such as [Patchew](https://patchew.org/QEMU/) and
+[Shippable](https://app.shippable.com/github/qemu/qemu/dashboard) but
+any developer with a docker setup can run the exact same images. For
+example if you want to check your patches won't break when compiled on
+a 32 bit ARM system you can run:
+
+    make docker-test-build@debian-armhf-cross J=n
+
+instead of tracking down a piece of ARM hardware to actually build on.
+Run `make docker` in your source tree to see the range of builds and
+tests it can support.
+
+# make check-tcg
+
+With the latest work [merged into
+master](https://git.qemu.org/?p=qemu.git;a=commit;h=de44c044420d1139480fa50c2d5be19223391218) we can now
+take advantage of both hand configured and docker based cross
+compilers to build test cases for TCG again. To run the TCG tests
+after you have built QEMU:
+
+    make check-tcg
+
+and the build system will build and run all the tests it can for your
+configured targets.
+
+# Rules for tests/tcg
+
+So now we have the infrastructure in place to add new tests what rules
+need to be followed to add new tests? 
+
+Well the first thing to note is currently all the tests are for
+linux-user versions of QEMU. This means the tests are all currently
+user-space tests that have access to the Linux syscall ABI.
+
+Another thing to note is the tests are standalone from the rest of the
+QEMU test infrastructure. To keep things simple they are compiled as
+standalone "static" binaries. As the cross-compilation setup can be
+quite rudimentary for some of the rarer architectures we only compile
+against a standard libc. There is no support for linking to other
+libraries like for example glib. Thread and maths support is part of
+glibc so shouldn't be a problem.
+
+Finally when writing new tests consider if it really is architecture
+specific or can be added to `tests/tcg/multiarch`. The multiarch tests
+are re-built for every supported architecture and should be the
+default place for anything explicitly testing syscalls and other
+common parts of the code base.
+
+# What's Next
+
+My hope with this work is we can start adding more tests to
+systematically defend functionality in linux-user. In fact I hope the
+first port of call to reproducing a crash would be writing a test case
+that can be added to our growing library of tests.
+
+Another thing that needs sorting out is getting toolchains for all of
+the less common architectures. The current work relies heavily on the
+excellent work of the Debian toolchain team in making multiarch
+aware cross compilers available in their distribution. However QEMU
+supports a lot more architectures than QEMU, some only as system
+emulations. In principle supporting them is as easy as adding another
+docker recipe but it might be these recipes end up having to compile
+the compilers from source.
+
+The `tests/tcg` directory still contains a number of source files we
+don't build. 
+
+The cris and openrisc directories contain user-space tests which just
+need the support of a toolchain and the relevant Makefile plumbing to
+be added.
+
+The lm32, mips and xtensa targets have a set of tests that need a
+system emulator. Aside from adding the compilers as docker images some
+additional work is needed to handle the differences between plain
+linux-user tests which can simply return an exit code to getting the
+results from a qemu-system emulation. Some architectures have
+semi-hosting support already for this while others report their test
+status over a simple serial link which will need to be parsed and
+handled in custom versions of the
+[`run-%:`](https://git.qemu.org/?p=qemu.git;a=blob;f=tests/tcg/Makefile;h=bf064153900a438e4ad8e2d79eaaac8a27d66062;hb=HEAD#l95)
+rule.
+
+