| Message ID | 20180621184116.910-1-alex.bennee@linaro.org |
|---|---|
| State | Superseded |
| Headers | show |
| Series | blogposts: add post about the new check-tcg infrastructure | expand |
Hi,
This series seems to have some coding style problems. See output below for
more information:
Type: series
Message-id: 20180621184116.910-1-alex.bennee@linaro.org
Subject: [Qemu-devel] [PATCH] blogposts: add post about the new check-tcg infrastructure
=== TEST SCRIPT BEGIN ===
#!/bin/bash
BASE=base
n=1
total=$(git log --oneline $BASE.. | wc -l)
failed=0
git config --local diff.renamelimit 0
git config --local diff.renames True
git config --local diff.algorithm histogram
commits="$(git log --format=%H --reverse $BASE..)"
for c in $commits; do
echo "Checking PATCH $n/$total: $(git log -n 1 --format=%s $c)..."
if ! git show $c --format=email | ./scripts/checkpatch.pl --mailback -; then
failed=1
echo
fi
n=$((n+1))
done
exit $failed
=== TEST SCRIPT END ===
Updating 3c8cf5a9c21ff8782164d1def7f44bd888713384
From https://github.com/patchew-project/qemu
* [new tag] patchew/20180621184116.910-1-alex.bennee@linaro.org -> patchew/20180621184116.910-1-alex.bennee@linaro.org
Switched to a new branch 'test'
0a765327ad blogposts: add post about the new check-tcg infrastructure
=== OUTPUT BEGIN ===
Checking PATCH 1/1: blogposts: add post about the new check-tcg infrastructure...
WARNING: added, moved or deleted file(s), does MAINTAINERS need updating?
#13:
new file mode 100644
ERROR: trailing whitespace
#102: FILE: _posts/2018-06-21-tcg-testing.md:85:
+need to be followed to add new tests? $
total: 1 errors, 1 warnings, 129 lines checked
Your patch has style problems, please review. If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
=== OUTPUT END ===
Test command exited with code: 1
---
Email generated automatically by Patchew [http://patchew.org/].
Please send your feedback to patchew-devel@redhat.com
On Thu, Jun 21, 2018 at 11:41 AM, Alex Bennée <alex.bennee@linaro.org> wrote: > Signed-off-by: Alex Bennée <alex.bennee@linaro.org> > --- > _posts/2018-06-21-tcg-testing.md | 129 +++++++++++++++++++++++++++++++ > 1 file changed, 129 insertions(+) > create mode 100644 _posts/2018-06-21-tcg-testing.md > > diff --git a/_posts/2018-06-21-tcg-testing.md b/_posts/2018-06-21-tcg-testing.md [...] > +The `tests/tcg` directory still contains a number of source files we > +don't build. Notably the cris, lm32, mips, openrisc and xtensa targets have > +a set of tests that need a system emulator. Now we have the > +infrastructure for compiling I hope we can get support for system > +tests added fairly quickly. There will need to be some work to figure > +out a nice common way to pass results back to the build-system. For > +linux-user this is simple as all programs can simply return their exit > +code however for system emulation this is a little more involved. xtensa tests pass exit codes to the build system through semihosting calls. If any of them fails make check fails as well. -- Thanks. -- Max
On 06/21/2018 06:12 PM, no-reply@patchew.org wrote: > This series seems to have some coding style problems. See output below for > more information: > > Type: series > Message-id: 20180621184116.910-1-alex.bennee@linaro.org > Subject: [Qemu-devel] [PATCH] blogposts: add post about the new check-tcg infrastructure [...]> === OUTPUT BEGIN === > Checking PATCH 1/1: blogposts: add post about the new check-tcg infrastructure... > WARNING: added, moved or deleted file(s), does MAINTAINERS need updating? > #13: > new file mode 100644 > > ERROR: trailing whitespace > #102: FILE: _posts/2018-06-21-tcg-testing.md:85: > +need to be followed to add new tests? $ =) > > total: 1 errors, 1 warnings, 129 lines checked > > Your patch has style problems, please review. If any of these errors > are false positives report them to the maintainer, see > CHECKPATCH in MAINTAINERS.
Max Filippov <jcmvbkbc@gmail.com> writes: > On Thu, Jun 21, 2018 at 11:41 AM, Alex Bennée <alex.bennee@linaro.org> wrote: >> Signed-off-by: Alex Bennée <alex.bennee@linaro.org> >> --- >> _posts/2018-06-21-tcg-testing.md | 129 +++++++++++++++++++++++++++++++ >> 1 file changed, 129 insertions(+) >> create mode 100644 _posts/2018-06-21-tcg-testing.md >> >> diff --git a/_posts/2018-06-21-tcg-testing.md b/_posts/2018-06-21-tcg-testing.md > > [...] > >> +The `tests/tcg` directory still contains a number of source files we >> +don't build. Notably the cris, lm32, mips, openrisc and xtensa targets have >> +a set of tests that need a system emulator. Now we have the >> +infrastructure for compiling I hope we can get support for system >> +tests added fairly quickly. There will need to be some work to figure >> +out a nice common way to pass results back to the build-system. For >> +linux-user this is simple as all programs can simply return their exit >> +code however for system emulation this is a little more involved. > > xtensa tests pass exit codes to the build system through semihosting calls. > If any of them fails make check fails as well. I've re-written that section as: The `tests/tcg` directory still contains a number of source files we don't build. The cris and openrisc directories contain user-space tests which just need the support of a toolchain and the relevant Makefile plumbing to be added. The lm32, mips and xtensa targets have a set of tests that need a system emulator. Aside from adding the compilers as docker images some additional work is needed to handle the differences between plain linux-user tests which can simply return an exit code to getting the results from a qemu-system emulation. Some architectures have semi-hosting support already for this while others report their test status over a simple serial link which will need to be parsed and handled in the `run-%:` test rule. How is that? Any chance you could look into what it would take to package up the xtensa toolchain in a docker container? Are they simply tarballs of binaries? -- Alex Bennée
On Fri, Jun 22, 2018 at 2:03 AM, Alex Bennée <alex.bennee@linaro.org> wrote: > > Max Filippov <jcmvbkbc@gmail.com> writes: > >> On Thu, Jun 21, 2018 at 11:41 AM, Alex Bennée <alex.bennee@linaro.org> wrote: >>> Signed-off-by: Alex Bennée <alex.bennee@linaro.org> >>> --- >>> _posts/2018-06-21-tcg-testing.md | 129 +++++++++++++++++++++++++++++++ >>> 1 file changed, 129 insertions(+) >>> create mode 100644 _posts/2018-06-21-tcg-testing.md >>> >>> diff --git a/_posts/2018-06-21-tcg-testing.md b/_posts/2018-06-21-tcg-testing.md >> >> [...] >> >>> +The `tests/tcg` directory still contains a number of source files we >>> +don't build. Notably the cris, lm32, mips, openrisc and xtensa targets have >>> +a set of tests that need a system emulator. Now we have the >>> +infrastructure for compiling I hope we can get support for system >>> +tests added fairly quickly. There will need to be some work to figure >>> +out a nice common way to pass results back to the build-system. For >>> +linux-user this is simple as all programs can simply return their exit >>> +code however for system emulation this is a little more involved. >> >> xtensa tests pass exit codes to the build system through semihosting calls. >> If any of them fails make check fails as well. > > I've re-written that section as: > > The `tests/tcg` directory still contains a number of source files we > don't build. > > The cris and openrisc directories contain user-space tests which just > need the support of a toolchain and the relevant Makefile plumbing to > be added. > > The lm32, mips and xtensa targets have a set of tests that need a > system emulator. Aside from adding the compilers as docker images some > additional work is needed to handle the differences between plain > linux-user tests which can simply return an exit code to getting the > results from a qemu-system emulation. Some architectures have > semi-hosting support already for this while others report their test > status over a simple serial link which will need to be parsed and > handled in the `run-%:` test rule. > > How is that? 'run-%' goal is only present in xtensa Makefile, other test suites have explicit loop in the 'check' goal. Otherwise LGTM. > Any chance you could look into what it would take to package up the > xtensa toolchain in a docker container? Can you point me to an example? > Are they simply tarballs of binaries? Yes, we have that option: https://github.com/foss-xtensa/toolchain/releases Or they may be built from source. -- Thanks. -- Max
Max Filippov <jcmvbkbc@gmail.com> writes: > On Fri, Jun 22, 2018 at 2:03 AM, Alex Bennée <alex.bennee@linaro.org> wrote: >> >> Max Filippov <jcmvbkbc@gmail.com> writes: >> >>> On Thu, Jun 21, 2018 at 11:41 AM, Alex Bennée <alex.bennee@linaro.org> wrote: >>>> Signed-off-by: Alex Bennée <alex.bennee@linaro.org> >>>> --- >>>> _posts/2018-06-21-tcg-testing.md | 129 +++++++++++++++++++++++++++++++ >>>> 1 file changed, 129 insertions(+) >>>> create mode 100644 _posts/2018-06-21-tcg-testing.md >>>> >>>> diff --git a/_posts/2018-06-21-tcg-testing.md b/_posts/2018-06-21-tcg-testing.md >>> >>> [...] >>> >>>> +The `tests/tcg` directory still contains a number of source files we >>>> +don't build. Notably the cris, lm32, mips, openrisc and xtensa targets have >>>> +a set of tests that need a system emulator. Now we have the >>>> +infrastructure for compiling I hope we can get support for system >>>> +tests added fairly quickly. There will need to be some work to figure >>>> +out a nice common way to pass results back to the build-system. For >>>> +linux-user this is simple as all programs can simply return their exit >>>> +code however for system emulation this is a little more involved. >>> >>> xtensa tests pass exit codes to the build system through semihosting calls. >>> If any of them fails make check fails as well. >> >> I've re-written that section as: >> >> The `tests/tcg` directory still contains a number of source files we >> don't build. >> >> The cris and openrisc directories contain user-space tests which just >> need the support of a toolchain and the relevant Makefile plumbing to >> be added. >> >> The lm32, mips and xtensa targets have a set of tests that need a >> system emulator. Aside from adding the compilers as docker images some >> additional work is needed to handle the differences between plain >> linux-user tests which can simply return an exit code to getting the >> results from a qemu-system emulation. Some architectures have >> semi-hosting support already for this while others report their test >> status over a simple serial link which will need to be parsed and >> handled in the `run-%:` test rule. >> >> How is that? > > 'run-%' goal is only present in xtensa Makefile, other test suites have > explicit loop in the 'check' goal. Otherwise LGTM. The TCG tests added a generic run-% rule: run-%: % $(call run-test, $<, $(QEMU) $<, "$< on $(TARGET_NAME)") Although individual tests can override it. > >> Any chance you could look into what it would take to package up the >> xtensa toolchain in a docker container? > > Can you point me to an example? Well this was the base attempt: https://github.com/stsquad/qemu/commit/daee9dc013b13b13e81f720176c5e379c05ed0e0 So I guess we need to add the overlay to the image? > >> Are they simply tarballs of binaries? > > Yes, we have that option: > > https://github.com/foss-xtensa/toolchain/releases > > Or they may be built from source. -- Alex Bennée
diff --git a/_posts/2018-06-21-tcg-testing.md b/_posts/2018-06-21-tcg-testing.md new file mode 100644 index 0000000..85abb7a --- /dev/null +++ b/_posts/2018-06-21-tcg-testing.md @@ -0,0 +1,129 @@ +--- +layout: post +title: "QEMU TCG Tests" +date: 2018-06-21 10:30:00:00 +0000 +last_modified_at: 2018-06-21 10:30:00:00 +0000 +author: Alex Bennée +categories: [testing, docker, compilation, tcg] +--- + +Ever since I started working on QEMU there was a small directory +called tests/tcg that was in a perpetually broken state. It contains +tests that exercise QEMU's ability to work across architectures using +the power of the Tiny Code Generator. However as these tests needed to +be compiled for the *guest* architectures and not the *host* +architecture - known as cross-compiling - most developers never ran +them. As the tests were hardly ever built inevitably a certain amount +of bit-rot set in. + +# Cross Compilers + +In the old days cross-compilation setups were almost all hand-crafted +affairs which involved building versions of binutils, gcc and a basic +libc. If you couldn't get someone to give you a pre-built tarball it +was something you laboured through once and hopefully never had to +touch again. There were even dedicated scripts like crosstool-ng which +attempted to make the process of patching and configuring your +toolchain easier. + +While the distributions have improved their support for cross +compilers over the years there are still plenty of variations in how +they are deployed. It is hard for a project like QEMU which has to +build on a wide range of operating systems and architectures to +seamlessly use any given distributions compiler setup. However for +those with cross compilers to hand `configure` now accepts two +additional flags: + + --cross-cc-$(ARCH) + --cross-cc-flags-$(ARCH) + +With a compiler specified for each guest architecture you want to test +the build system can now build and run the tests. However for +developers that don't have cross compilers around they can now take +advantage of QEMU's docker images. + +# Enter Docker Containers + +If you work in IT you would be hard pressed not to have noticed the +hype around Docker and the concept of containerisation over the last +few years. Put simply containers allow you to define a known working +set of software that gets run in an isolated environment for a given +task. While this has many uses for QEMU it allows us to define build +environments that any developer can run without having to mess around +with their preferred host setup. + +Over the last few years QEMU's build system has been expanding the +number of docker images it supports. Most of this has been in service +of our CI testing such as [Patchew](https://patchew.org/QEMU/) and +[Shippable](https://app.shippable.com/github/qemu/qemu/dashboard) but +any developer with a docker setup can run the exact same images. For +example if you want to check your patches won't break when compiled on +a 32 bit ARM system you can run: + + make docker-test-build@debian-armhf-cross J=n + +instead of tracking down a piece of ARM hardware to actually build on. +Run `make docker` in your source tree to see the range of builds and +tests it can support. + +# make check-tcg + +With the latest work [merged into +master](https://git.qemu.org/?p=qemu.git;a=commit;h=TODO) we can now +take advantage of both hand configured and docker based cross +compilers to build test cases for TCG again. To run the TCG tests +after you have built QEMU: + + make check-tcg + +and the build system will build and run all the tests it can for your +configured targets. + +# Rules for tests/tcg + +So now we have the infrastructure in place to add new tests what rules +need to be followed to add new tests? + +Well the first thing to note is currently all the tests are for +linux-user versions of QEMU. This means the tests are all currently +user-space tests that have access to the Linux syscall ABI. + +Another thing to note is the tests are standalone from the rest of the +QEMU test infrastructure. To keep things simple they are compiled as +standalone "static" binaries. As the cross-compilation setup can be +quite rudimentary for some of the rarer architectures we only compile +against a standard libc. There is no support for linking to other +libraries like for example glib. Thread and maths support is part of +glibc so shouldn't be a problem. + +Finally when writing new tests consider if it really is architecture +specific or can be added to `tests/tcg/multiarch`. The multiarch tests +are re-built for every supported architecture and should be the +default place for anything explicitly testing syscalls and other +common parts of the code base. + +# What's Next + +My hope with this work is we can start adding more tests to +systematically defend functionality in linux-user. In fact I hope the +first port of call to reproducing a crash would be writing a test case +that can be added to our growing library of tests. + +Another thing that needs sorting out is getting toolchains for all of +the less common architectures. The current work relies heavily on the +excellent work of the Debian toolchain team in making multiarch +aware cross compilers available in their distribution. However QEMU +supports a lot more architectures than QEMU, some only as system +emulations. In principle supporting them is as easy as adding another +docker recipe but it might be these recipes end up having to compile +the compilers from source. + +The `tests/tcg` directory still contains a number of source files we +don't build. Notably the cris, lm32, mips, openrisc and xtensa targets have +a set of tests that need a system emulator. Now we have the +infrastructure for compiling I hope we can get support for system +tests added fairly quickly. There will need to be some work to figure +out a nice common way to pass results back to the build-system. For +linux-user this is simple as all programs can simply return their exit +code however for system emulation this is a little more involved. +
Signed-off-by: Alex Bennée <alex.bennee@linaro.org> --- _posts/2018-06-21-tcg-testing.md | 129 +++++++++++++++++++++++++++++++ 1 file changed, 129 insertions(+) create mode 100644 _posts/2018-06-21-tcg-testing.md -- 2.17.1