| Message ID | 20160920135616.2215-1-alex.bennee@linaro.org |
|---|---|
| State | New |
| Headers | show |
Fam Zheng <famz@redhat.com> writes: > On Tue, 09/20 14:56, Alex Bennée wrote: >> This re-factors the docker makefile to include a docker-run target which >> can be controlled entirely from environment variables specified on the >> make command line. This allows us to run against any given docker image >> we may have in our repository, for example: >> >> make docker-run TEST="test-quick" IMAGE="debian:arm64" \ >> EXECUTABLE=./aarch64-linux-user/qemu-aarch64 >> >> The existing docker-foo@bar targets still work but the inline >> verification has been shunted into other target prerequisites before a >> sub-make is invoked for the docker-run target. > > Hi Alex, > > I understand sometimes one can have specialized images, but still: is it > possible to convert them to Dockerfile and include in the tree? > > Or, is this for testing/debugging purpose? A bit of both. In this particular use case I'm using a debootstrap image while updating the binfmt_misc executable. Currently there is a 1->N relationship for debootstrap as we can bootstrap multiple architectures in different images. By splitting the docker-run from the expansions we give ourselves a little more flexibility for running stuff. But I think it's also useful for testing/debugging. I wrote this up as I was trying to debug a Travis build failure with gcc-6 so I was generating lots of test images and wanting to build against those. I would also like to add a travis Dockerfile at some point but at the moment what exactly goes into one of those is a little opaque to me. > >> >> Signed-off-by: Alex Bennée <alex.bennee@linaro.org> >> >> --- >> NB: I dropped the awk magic that verifies the image exists before >> running. I couldn't get the thing to work in my shell so wasn't quite >> sure what it was doing. > > It was to allow "make docker-test" to skip debian-bootstrap image if it is not > there (e.g. when qemu-user not available). Ahh ok. I got a little confused as the docker images command can filter things based on tag so maybe we can come up with a cleaner test? > > I'm not much too concerned about that though, since most of the time we will > use docker-FOO@BAR, for specific combinations, instead of docker-test for a > blanket coverage. What does patchew use? > >> --- >> tests/docker/Makefile.include | 82 ++++++++++++++++++++++++++++++++----------- >> 1 file changed, 62 insertions(+), 20 deletions(-) >> >> diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include >> index 19d4cc7..ee47085 100644 >> --- a/tests/docker/Makefile.include >> +++ b/tests/docker/Makefile.include >> @@ -101,31 +101,73 @@ docker: >> @echo ' NOCACHE=1 Ignore cache when build images.' >> @echo ' EXECUTABLE=<path> Include executable in image.' >> >> -docker-run-%: CMD = $(shell echo '$@' | sed -e 's/docker-run-\([^@]*\)@\(.*\)/\1/') >> -docker-run-%: IMAGE = $(shell echo '$@' | sed -e 's/docker-run-\([^@]*\)@\(.*\)/\2/') >> -docker-run-%: docker-qemu-src >> +# This rule if for directly running against an arbitary docker target. >> +# It is called by the expanded docker targets (e.g. make >> +# docker-test-foo@bar) which will do additional verification. >> +# >> +# For example: make docker-run TEST="test-quick" IMAGE="debian:arm64" EXECUTABLE=./aarch64-linux-user/qemu-aarch64 >> +# >> +docker-run: docker-qemu-src > > This target should probably be documented in "make docker". OK > >> @mkdir -p "$(DOCKER_CCACHE_DIR)" >> - @if test -z "$(IMAGE)" || test -z "$(CMD)"; \ >> - then echo "Invalid target"; exit 1; \ >> + @if test -z "$(IMAGE)" || test -z "$(TEST)"; \ >> + then echo "Invalid target $(IMAGE)/$(TEST)"; exit 1; \ >> fi >> - $(if $(filter $(TESTS),$(CMD)),$(if $(filter $(IMAGES),$(IMAGE)), \ >> - $(call quiet-command,\ >> - if $(SRC_PATH)/tests/docker/docker.py images | \ >> - awk '$$1=="qemu" && $$2=="$(IMAGE)"{found=1} END{exit(!found)}'; then \ >> - $(SRC_PATH)/tests/docker/docker.py run $(if $V,,--rm) \ >> - -t \ >> - $(if $(DEBUG),-i,--net=none) \ >> - -e TARGET_LIST=$(TARGET_LIST) \ >> + $(if $(EXECUTABLE), \ >> + $(call quiet-command, \ >> + $(SRC_PATH)/tests/docker/docker.py update \ >> + $(IMAGE) $(EXECUTABLE))) >> + $(call quiet-command, \ >> + $(SRC_PATH)/tests/docker/docker.py run \ >> + -t \ >> + $(if $V,,--rm) \ >> + $(if $(DEBUG),-i,--net=none) \ >> + -e TARGET_LIST=$(TARGET_LIST) \ >> -e EXTRA_CONFIGURE_OPTS=$(EXTRA_CONFIGURE_OPTS) \ >> - -e V=$V -e J=$J -e DEBUG=$(DEBUG)\ >> - -e CCACHE_DIR=/var/tmp/ccache \ >> + -e V=$V -e J=$J -e DEBUG=$(DEBUG) \ >> + -e CCACHE_DIR=/var/tmp/ccache \ >> -v $$(readlink -e $(DOCKER_SRC_COPY)):/var/tmp/qemu:z$(COMMA)ro \ >> -v $(DOCKER_CCACHE_DIR):/var/tmp/ccache:z \ >> - qemu:$(IMAGE) \ >> - /var/tmp/qemu/run \ >> - $(CMD); \ >> - fi \ >> - , " RUN $(CMD) in $(IMAGE)"))) >> + $(IMAGE) \ >> + /var/tmp/qemu/run \ >> + $(TEST), " RUN $(TEST) in ${IMAGE}") >> + >> +# >> +# Verification targets >> +# >> +# These targets help verify the test (CMD) and docker tag (IMAGE) are >> +# part of the built in set of tests and images. You can still call the >> +# docker-run target directly for testsing against arbitary images. > > s/arbitary/arbitrary/ > > One more of this below. OK. > >> +# >> + >> +docker-verify-image-%: IMAGE = $(shell echo '$@' | sed -e 's/docker-verify-image-\([^@]*\)@\(.*\)/\2/') >> +docker-verify-image-%: >> + @if test -z "$(IMAGE)"; \ >> + then echo "Invalid image"; exit 1; \ >> + fi >> + $(if $(findstring $(IMAGE), $(DOCKER_IMAGES)) \ > > This seems inaccurate. what if it is in a form of "$(findstring abc, > abcde)"? I'll see if I can replace it with the original filter test. > >> + , @echo "Verified $(IMAGE)" \ >> + , @echo "$(IMAGE) is not a known image"; exit 1) >> + >> +docker-verify-test-%: CMD = $(shell echo '$@' | sed -e 's/docker-verify-test-\([^@]*\)@\(.*\)/\1/') >> +docker-verify-test-%: >> + @if test -z "$(CMD)"; \ >> + then echo "Invalid test"; exit 1; \ >> + fi >> + $(if $(findstring $(CMD), $(DOCKER_TESTS)) \ > > The same question to findstring as above. > >> + , @echo "Verified $(CMD)" \ > > No need to echo anything in case of verified, or at least use quiet-command > please. Sure. This was mostly convincing me I got my make-fu right ;-) > >> + , @echo "$(CMD) is not a known test"; exit 1) >> + >> +# Run targets >> +# >> +# This will take a target such as docker-test-foo@bar and verify that: >> +# - the test test-foo is a known test >> +# - the image bar is a known image >> +# >> +# It will then call the docker-run >> +docker-run-%: CMD = $(shell echo '$@' | sed -e 's/docker-run-\([^@]*\)@\(.*\)/\1/') >> +docker-run-%: IMAGE = $(shell echo '$@' | sed -e 's/docker-run-\([^@]*\)@\(.*\)/\2/') >> +docker-run-%: docker-verify-image-% docker-verify-test-% >> + @make docker-run TEST=$(CMD) IMAGE=qemu:$(IMAGE) >> >> docker-clean: >> $(call quiet-command, $(SRC_PATH)/tests/docker/docker.py clean) >> -- >> 2.9.3 >> > > Fam -- Alex Bennée
Fam Zheng <famz@redhat.com> writes: > On Wed, 09/21 08:50, Alex Bennée wrote: >> >> Fam Zheng <famz@redhat.com> writes: >> >> > On Tue, 09/20 14:56, Alex Bennée wrote: >> >> This re-factors the docker makefile to include a docker-run target which >> >> can be controlled entirely from environment variables specified on the >> >> make command line. This allows us to run against any given docker image >> >> we may have in our repository, for example: >> >> >> >> make docker-run TEST="test-quick" IMAGE="debian:arm64" \ >> >> EXECUTABLE=./aarch64-linux-user/qemu-aarch64 >> >> >> >> The existing docker-foo@bar targets still work but the inline >> >> verification has been shunted into other target prerequisites before a >> >> sub-make is invoked for the docker-run target. >> > >> > Hi Alex, >> > >> > I understand sometimes one can have specialized images, but still: is it >> > possible to convert them to Dockerfile and include in the tree? >> > >> > Or, is this for testing/debugging purpose? >> >> A bit of both. In this particular use case I'm using a debootstrap image >> while updating the binfmt_misc executable. Currently there is a 1->N >> relationship for debootstrap as we can bootstrap multiple architectures >> in different images. By splitting the docker-run from the expansions we >> give ourselves a little more flexibility for running stuff. >> >> But I think it's also useful for testing/debugging. I wrote this up as I >> was trying to debug a Travis build failure with gcc-6 so I was >> generating lots of test images and wanting to build against those. I >> would also like to add a travis Dockerfile at some point but at the >> moment what exactly goes into one of those is a little opaque to me. > > Thanks for clarifying, and I agree this feature is really nice in general. > >> >> > >> >> >> >> Signed-off-by: Alex Bennée <alex.bennee@linaro.org> >> >> >> >> --- >> >> NB: I dropped the awk magic that verifies the image exists before >> >> running. I couldn't get the thing to work in my shell so wasn't quite >> >> sure what it was doing. >> > >> > It was to allow "make docker-test" to skip debian-bootstrap image if it is not >> > there (e.g. when qemu-user not available). >> >> Ahh ok. I got a little confused as the docker images command can filter >> things based on tag so maybe we can come up with a cleaner test? > > For once it used a format option of "docker images" that isn't available on > RHEL 7, per requested I changed it to the unobvious awk test. > >> >> > >> > I'm not much too concerned about that though, since most of the time we will >> > use docker-FOO@BAR, for specific combinations, instead of docker-test for a >> > blanket coverage. >> >> What does patchew use? > > The general strategy of patchew is good coverage of both tests and images, > without multiplexing them which could make testing one patch infinitely long on > a simple minded tester. > > For now, we have: > > docker-test-quick@centos6 > docker-test-mingw@fedora > > And staging (pending because of some mysterious false positives): > > docker-test-quick@min-glib > > I also plan to extend to centos7 and ubuntu in the middle term, and give cross > compiling for OSX a try in the long run (googling says it's technically > possible). FWIW we already have some coverage of the MacOSX builds via Travis (although being able to run it quickly on a dev system would be useful). > > I haven't prioritied debootstrap for now, because arm is not too different than > x86 in terms of endianness and stuff, and qemu-user is probably much slower > than native compilers. It is much slower although qemu-user can at least take advantage of all those extra cores on your server ;-) 32 bit builds are also an area that needs good coverage as I'm pretty sure most devs have only x86_64 boxes these days. > > But still, BE images will be a compelling reason, if there comes one. > > Fam -- Alex Bennée
Fam Zheng <famz@redhat.com> writes: > On Wed, 09/21 10:44, Alex Bennée wrote: >> >> FWIW we already have some coverage of the MacOSX builds via Travis >> (although being able to run it quickly on a dev system would be useful). > > Being able to detect breakage earlier than a pull req bounce from Peter would > also be good. True. Although more people have started enabling Travis on their private git trees, we just don't see it as much as IRC doesn't get spammed for anything other than runs against the official tree. > >> >> > >> > I haven't prioritied debootstrap for now, because arm is not too different than >> > x86 in terms of endianness and stuff, and qemu-user is probably much slower >> > than native compilers. >> >> It is much slower although qemu-user can at least take advantage of all >> those extra cores on your server ;-) >> >> 32 bit builds are also an area that needs good coverage as I'm pretty >> sure most devs have only x86_64 boxes these days. > > Yes, test-mingw has it. Ahh cool :-) -- Alex Bennée
diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include index 19d4cc7..ee47085 100644 --- a/tests/docker/Makefile.include +++ b/tests/docker/Makefile.include @@ -101,31 +101,73 @@ docker: @echo ' NOCACHE=1 Ignore cache when build images.' @echo ' EXECUTABLE=<path> Include executable in image.' -docker-run-%: CMD = $(shell echo '$@' | sed -e 's/docker-run-\([^@]*\)@\(.*\)/\1/') -docker-run-%: IMAGE = $(shell echo '$@' | sed -e 's/docker-run-\([^@]*\)@\(.*\)/\2/') -docker-run-%: docker-qemu-src +# This rule if for directly running against an arbitary docker target. +# It is called by the expanded docker targets (e.g. make +# docker-test-foo@bar) which will do additional verification. +# +# For example: make docker-run TEST="test-quick" IMAGE="debian:arm64" EXECUTABLE=./aarch64-linux-user/qemu-aarch64 +# +docker-run: docker-qemu-src @mkdir -p "$(DOCKER_CCACHE_DIR)" - @if test -z "$(IMAGE)" || test -z "$(CMD)"; \ - then echo "Invalid target"; exit 1; \ + @if test -z "$(IMAGE)" || test -z "$(TEST)"; \ + then echo "Invalid target $(IMAGE)/$(TEST)"; exit 1; \ fi - $(if $(filter $(TESTS),$(CMD)),$(if $(filter $(IMAGES),$(IMAGE)), \ - $(call quiet-command,\ - if $(SRC_PATH)/tests/docker/docker.py images | \ - awk '$$1=="qemu" && $$2=="$(IMAGE)"{found=1} END{exit(!found)}'; then \ - $(SRC_PATH)/tests/docker/docker.py run $(if $V,,--rm) \ - -t \ - $(if $(DEBUG),-i,--net=none) \ - -e TARGET_LIST=$(TARGET_LIST) \ + $(if $(EXECUTABLE), \ + $(call quiet-command, \ + $(SRC_PATH)/tests/docker/docker.py update \ + $(IMAGE) $(EXECUTABLE))) + $(call quiet-command, \ + $(SRC_PATH)/tests/docker/docker.py run \ + -t \ + $(if $V,,--rm) \ + $(if $(DEBUG),-i,--net=none) \ + -e TARGET_LIST=$(TARGET_LIST) \ -e EXTRA_CONFIGURE_OPTS=$(EXTRA_CONFIGURE_OPTS) \ - -e V=$V -e J=$J -e DEBUG=$(DEBUG)\ - -e CCACHE_DIR=/var/tmp/ccache \ + -e V=$V -e J=$J -e DEBUG=$(DEBUG) \ + -e CCACHE_DIR=/var/tmp/ccache \ -v $$(readlink -e $(DOCKER_SRC_COPY)):/var/tmp/qemu:z$(COMMA)ro \ -v $(DOCKER_CCACHE_DIR):/var/tmp/ccache:z \ - qemu:$(IMAGE) \ - /var/tmp/qemu/run \ - $(CMD); \ - fi \ - , " RUN $(CMD) in $(IMAGE)"))) + $(IMAGE) \ + /var/tmp/qemu/run \ + $(TEST), " RUN $(TEST) in ${IMAGE}") + +# +# Verification targets +# +# These targets help verify the test (CMD) and docker tag (IMAGE) are +# part of the built in set of tests and images. You can still call the +# docker-run target directly for testsing against arbitary images. +# + +docker-verify-image-%: IMAGE = $(shell echo '$@' | sed -e 's/docker-verify-image-\([^@]*\)@\(.*\)/\2/') +docker-verify-image-%: + @if test -z "$(IMAGE)"; \ + then echo "Invalid image"; exit 1; \ + fi + $(if $(findstring $(IMAGE), $(DOCKER_IMAGES)) \ + , @echo "Verified $(IMAGE)" \ + , @echo "$(IMAGE) is not a known image"; exit 1) + +docker-verify-test-%: CMD = $(shell echo '$@' | sed -e 's/docker-verify-test-\([^@]*\)@\(.*\)/\1/') +docker-verify-test-%: + @if test -z "$(CMD)"; \ + then echo "Invalid test"; exit 1; \ + fi + $(if $(findstring $(CMD), $(DOCKER_TESTS)) \ + , @echo "Verified $(CMD)" \ + , @echo "$(CMD) is not a known test"; exit 1) + +# Run targets +# +# This will take a target such as docker-test-foo@bar and verify that: +# - the test test-foo is a known test +# - the image bar is a known image +# +# It will then call the docker-run +docker-run-%: CMD = $(shell echo '$@' | sed -e 's/docker-run-\([^@]*\)@\(.*\)/\1/') +docker-run-%: IMAGE = $(shell echo '$@' | sed -e 's/docker-run-\([^@]*\)@\(.*\)/\2/') +docker-run-%: docker-verify-image-% docker-verify-test-% + @make docker-run TEST=$(CMD) IMAGE=qemu:$(IMAGE) docker-clean: $(call quiet-command, $(SRC_PATH)/tests/docker/docker.py clean)
This re-factors the docker makefile to include a docker-run target which can be controlled entirely from environment variables specified on the make command line. This allows us to run against any given docker image we may have in our repository, for example: make docker-run TEST="test-quick" IMAGE="debian:arm64" \ EXECUTABLE=./aarch64-linux-user/qemu-aarch64 The existing docker-foo@bar targets still work but the inline verification has been shunted into other target prerequisites before a sub-make is invoked for the docker-run target. Signed-off-by: Alex Bennée <alex.bennee@linaro.org> --- NB: I dropped the awk magic that verifies the image exists before running. I couldn't get the thing to work in my shell so wasn't quite sure what it was doing. --- tests/docker/Makefile.include | 82 ++++++++++++++++++++++++++++++++----------- 1 file changed, 62 insertions(+), 20 deletions(-) -- 2.9.3