diff mbox

[RFC] test/docker/Makefile.include: add a generic docker-run target

Message ID 20160920135616.2215-1-alex.bennee@linaro.org
State New
Headers show

Commit Message

Alex Bennée Sept. 20, 2016, 1:56 p.m. UTC
This re-factors the docker makefile to include a docker-run target which
can be controlled entirely from environment variables specified on the
make command line. This allows us to run against any given docker image
we may have in our repository, for example:

    make docker-run TEST="test-quick" IMAGE="debian:arm64" \
         EXECUTABLE=./aarch64-linux-user/qemu-aarch64

The existing docker-foo@bar targets still work but the inline
verification has been shunted into other target prerequisites before a
sub-make is invoked for the docker-run target.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>


---
NB: I dropped the awk magic that verifies the image exists before
running. I couldn't get the thing to work in my shell so wasn't quite
sure what it was doing.
---
 tests/docker/Makefile.include | 82 ++++++++++++++++++++++++++++++++-----------
 1 file changed, 62 insertions(+), 20 deletions(-)

-- 
2.9.3

Comments

Alex Bennée Sept. 21, 2016, 7:50 a.m. UTC | #1
Fam Zheng <famz@redhat.com> writes:

> On Tue, 09/20 14:56, Alex Bennée wrote:

>> This re-factors the docker makefile to include a docker-run target which

>> can be controlled entirely from environment variables specified on the

>> make command line. This allows us to run against any given docker image

>> we may have in our repository, for example:

>>

>>     make docker-run TEST="test-quick" IMAGE="debian:arm64" \

>>          EXECUTABLE=./aarch64-linux-user/qemu-aarch64

>>

>> The existing docker-foo@bar targets still work but the inline

>> verification has been shunted into other target prerequisites before a

>> sub-make is invoked for the docker-run target.

>

> Hi Alex,

>

> I understand sometimes one can have specialized images, but still: is it

> possible to convert them to Dockerfile and include in the tree?

>

> Or, is this for testing/debugging purpose?


A bit of both. In this particular use case I'm using a debootstrap image
while updating the binfmt_misc executable. Currently there is a 1->N
relationship for debootstrap as we can bootstrap multiple architectures
in different images. By splitting the docker-run from the expansions we
give ourselves a little more flexibility for running stuff.

But I think it's also useful for testing/debugging. I wrote this up as I
was trying to debug a Travis build failure with gcc-6 so I was
generating lots of test images and wanting to build against those. I
would also like to add a travis Dockerfile at some point but at the
moment what exactly goes into one of those is a little opaque to me.

>

>>

>> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>

>>

>> ---

>> NB: I dropped the awk magic that verifies the image exists before

>> running. I couldn't get the thing to work in my shell so wasn't quite

>> sure what it was doing.

>

> It was to allow "make docker-test" to skip debian-bootstrap image if it is not

> there (e.g. when qemu-user not available).


Ahh ok. I got a little confused as the docker images command can filter
things based on tag so maybe we can come up with a cleaner test?

>

> I'm not much too concerned about that though, since most of the time we will

> use docker-FOO@BAR, for specific combinations, instead of docker-test for a

> blanket coverage.


What does patchew use?

>

>> ---

>>  tests/docker/Makefile.include | 82 ++++++++++++++++++++++++++++++++-----------

>>  1 file changed, 62 insertions(+), 20 deletions(-)

>>

>> diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include

>> index 19d4cc7..ee47085 100644

>> --- a/tests/docker/Makefile.include

>> +++ b/tests/docker/Makefile.include

>> @@ -101,31 +101,73 @@ docker:

>>  	@echo '    NOCACHE=1            Ignore cache when build images.'

>>  	@echo '    EXECUTABLE=<path>    Include executable in image.'

>>

>> -docker-run-%: CMD = $(shell echo '$@' | sed -e 's/docker-run-\([^@]*\)@\(.*\)/\1/')

>> -docker-run-%: IMAGE = $(shell echo '$@' | sed -e 's/docker-run-\([^@]*\)@\(.*\)/\2/')

>> -docker-run-%: docker-qemu-src

>> +# This rule if for directly running against an arbitary docker target.

>> +# It is called by the expanded docker targets (e.g. make

>> +# docker-test-foo@bar) which will do additional verification.

>> +#

>> +# For example: make docker-run TEST="test-quick" IMAGE="debian:arm64" EXECUTABLE=./aarch64-linux-user/qemu-aarch64

>> +#

>> +docker-run: docker-qemu-src

>

> This target should probably be documented in "make docker".


OK

>

>>  	@mkdir -p "$(DOCKER_CCACHE_DIR)"

>> -	@if test -z "$(IMAGE)" || test -z "$(CMD)"; \

>> -		then echo "Invalid target"; exit 1; \

>> +	@if test -z "$(IMAGE)" || test -z "$(TEST)"; \

>> +		then echo "Invalid target $(IMAGE)/$(TEST)"; exit 1; \

>>  	fi

>> -	$(if $(filter $(TESTS),$(CMD)),$(if $(filter $(IMAGES),$(IMAGE)), \

>> -		$(call quiet-command,\

>> -			if $(SRC_PATH)/tests/docker/docker.py images | \

>> -				awk '$$1=="qemu" && $$2=="$(IMAGE)"{found=1} END{exit(!found)}'; then \

>> -				$(SRC_PATH)/tests/docker/docker.py run $(if $V,,--rm) \

>> -				-t \

>> -				$(if $(DEBUG),-i,--net=none) \

>> -				-e TARGET_LIST=$(TARGET_LIST) \

>> +	$(if $(EXECUTABLE),						\

>> +		$(call quiet-command,					\

>> +			$(SRC_PATH)/tests/docker/docker.py update 	\

>> +			$(IMAGE) $(EXECUTABLE)))

>> +	$(call quiet-command,						\

>> +		$(SRC_PATH)/tests/docker/docker.py run 			\

>> +				-t 					\

>> +				$(if $V,,--rm) 				\

>> +				$(if $(DEBUG),-i,--net=none) 		\

>> +				-e TARGET_LIST=$(TARGET_LIST) 		\

>>  				-e EXTRA_CONFIGURE_OPTS=$(EXTRA_CONFIGURE_OPTS) \

>> -				-e V=$V -e J=$J -e DEBUG=$(DEBUG)\

>> -				-e CCACHE_DIR=/var/tmp/ccache \

>> +				-e V=$V -e J=$J -e DEBUG=$(DEBUG)	\

>> +				-e CCACHE_DIR=/var/tmp/ccache 		\

>>  				-v $$(readlink -e $(DOCKER_SRC_COPY)):/var/tmp/qemu:z$(COMMA)ro \

>>  				-v $(DOCKER_CCACHE_DIR):/var/tmp/ccache:z \

>> -				qemu:$(IMAGE) \

>> -				/var/tmp/qemu/run \

>> -				$(CMD); \

>> -			fi \

>> -			, "  RUN $(CMD) in $(IMAGE)")))

>> +				$(IMAGE) 				\

>> +				/var/tmp/qemu/run 			\

>> +				$(TEST), "  RUN $(TEST) in ${IMAGE}")

>> +

>> +#

>> +# Verification targets

>> +#

>> +# These targets help verify the test (CMD) and docker tag (IMAGE) are

>> +# part of the built in set of tests and images. You can still call the

>> +# docker-run target directly for testsing against arbitary images.

>

> s/arbitary/arbitrary/

>

> One more of this below.


OK.

>

>> +#

>> +

>> +docker-verify-image-%: IMAGE = $(shell echo '$@' | sed -e 's/docker-verify-image-\([^@]*\)@\(.*\)/\2/')

>> +docker-verify-image-%:

>> +	@if test -z "$(IMAGE)";			 	\

>> +		then echo "Invalid image"; exit 1; 	\

>> +	fi

>> +	$(if $(findstring $(IMAGE), $(DOCKER_IMAGES))	\

>

> This seems inaccurate. what if it is in a form of "$(findstring abc,

> abcde)"?


I'll see if I can replace it with the original filter test.

>

>> +		, @echo "Verified $(IMAGE)"		\

>> +		, @echo "$(IMAGE) is not a known image"; exit 1)

>> +

>> +docker-verify-test-%: CMD = $(shell echo '$@' | sed -e 's/docker-verify-test-\([^@]*\)@\(.*\)/\1/')

>> +docker-verify-test-%:

>> +	@if test -z "$(CMD)"; 				\

>> +		then echo "Invalid test"; exit 1; 	\

>> +	fi

>> +	$(if $(findstring $(CMD), $(DOCKER_TESTS))	\

>

> The same question to findstring as above.

>

>> +		, @echo "Verified $(CMD)"		\

>

> No need to echo anything in case of verified, or at least use quiet-command

> please.


Sure. This was mostly convincing me I got my make-fu right ;-)

>

>> +		, @echo "$(CMD) is not a known test"; exit 1)

>> +

>> +# Run targets

>> +#

>> +# This will take a target such as docker-test-foo@bar and verify that:

>> +#  - the test test-foo is a known test

>> +#  - the image bar is a known image

>> +#

>> +# It will then call the docker-run

>> +docker-run-%: CMD = $(shell echo '$@' | sed -e 's/docker-run-\([^@]*\)@\(.*\)/\1/')

>> +docker-run-%: IMAGE = $(shell echo '$@' | sed -e 's/docker-run-\([^@]*\)@\(.*\)/\2/')

>> +docker-run-%: docker-verify-image-% docker-verify-test-%

>> +	@make docker-run TEST=$(CMD) IMAGE=qemu:$(IMAGE)

>>

>>  docker-clean:

>>  	$(call quiet-command, $(SRC_PATH)/tests/docker/docker.py clean)

>> --

>> 2.9.3

>>

>

> Fam



--
Alex Bennée
Alex Bennée Sept. 21, 2016, 9:44 a.m. UTC | #2
Fam Zheng <famz@redhat.com> writes:

> On Wed, 09/21 08:50, Alex Bennée wrote:

>>

>> Fam Zheng <famz@redhat.com> writes:

>>

>> > On Tue, 09/20 14:56, Alex Bennée wrote:

>> >> This re-factors the docker makefile to include a docker-run target which

>> >> can be controlled entirely from environment variables specified on the

>> >> make command line. This allows us to run against any given docker image

>> >> we may have in our repository, for example:

>> >>

>> >>     make docker-run TEST="test-quick" IMAGE="debian:arm64" \

>> >>          EXECUTABLE=./aarch64-linux-user/qemu-aarch64

>> >>

>> >> The existing docker-foo@bar targets still work but the inline

>> >> verification has been shunted into other target prerequisites before a

>> >> sub-make is invoked for the docker-run target.

>> >

>> > Hi Alex,

>> >

>> > I understand sometimes one can have specialized images, but still: is it

>> > possible to convert them to Dockerfile and include in the tree?

>> >

>> > Or, is this for testing/debugging purpose?

>>

>> A bit of both. In this particular use case I'm using a debootstrap image

>> while updating the binfmt_misc executable. Currently there is a 1->N

>> relationship for debootstrap as we can bootstrap multiple architectures

>> in different images. By splitting the docker-run from the expansions we

>> give ourselves a little more flexibility for running stuff.

>>

>> But I think it's also useful for testing/debugging. I wrote this up as I

>> was trying to debug a Travis build failure with gcc-6 so I was

>> generating lots of test images and wanting to build against those. I

>> would also like to add a travis Dockerfile at some point but at the

>> moment what exactly goes into one of those is a little opaque to me.

>

> Thanks for clarifying, and I agree this feature is really nice in general.

>

>>

>> >

>> >>

>> >> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>

>> >>

>> >> ---

>> >> NB: I dropped the awk magic that verifies the image exists before

>> >> running. I couldn't get the thing to work in my shell so wasn't quite

>> >> sure what it was doing.

>> >

>> > It was to allow "make docker-test" to skip debian-bootstrap image if it is not

>> > there (e.g. when qemu-user not available).

>>

>> Ahh ok. I got a little confused as the docker images command can filter

>> things based on tag so maybe we can come up with a cleaner test?

>

> For once it used a format option of "docker images" that isn't available on

> RHEL 7, per requested I changed it to the unobvious awk test.

>

>>

>> >

>> > I'm not much too concerned about that though, since most of the time we will

>> > use docker-FOO@BAR, for specific combinations, instead of docker-test for a

>> > blanket coverage.

>>

>> What does patchew use?

>

> The general strategy of patchew is good coverage of both tests and images,

> without multiplexing them which could make testing one patch infinitely long on

> a simple minded tester.

>

> For now, we have:

>

>     docker-test-quick@centos6

>     docker-test-mingw@fedora

>

> And staging (pending because of some mysterious false positives):

>

>     docker-test-quick@min-glib

>

> I also plan to extend to centos7 and ubuntu in the middle term, and give cross

> compiling for OSX a try in the long run (googling says it's technically

> possible).


FWIW we already have some coverage of the MacOSX builds via Travis
(although being able to run it quickly on a dev system would be useful).

>

> I haven't prioritied debootstrap for now, because arm is not too different than

> x86 in terms of endianness and stuff, and qemu-user is probably much slower

> than native compilers.


It is much slower although qemu-user can at least take advantage of all
those extra cores on your server ;-)

32 bit builds are also an area that needs good coverage as I'm pretty
sure most devs have only x86_64 boxes these days.

>

> But still, BE images will be a compelling reason, if there comes one.

>

> Fam



--
Alex Bennée
Alex Bennée Sept. 21, 2016, 10:22 a.m. UTC | #3
Fam Zheng <famz@redhat.com> writes:

> On Wed, 09/21 10:44, Alex Bennée wrote:

>>

>> FWIW we already have some coverage of the MacOSX builds via Travis

>> (although being able to run it quickly on a dev system would be useful).

>

> Being able to detect breakage earlier than a pull req bounce from Peter would

> also be good.


True. Although more people have started enabling Travis on their private
git trees, we just don't see it as much as IRC doesn't get spammed for
anything other than runs against the official tree.

>

>>

>> >

>> > I haven't prioritied debootstrap for now, because arm is not too different than

>> > x86 in terms of endianness and stuff, and qemu-user is probably much slower

>> > than native compilers.

>>

>> It is much slower although qemu-user can at least take advantage of all

>> those extra cores on your server ;-)

>>

>> 32 bit builds are also an area that needs good coverage as I'm pretty

>> sure most devs have only x86_64 boxes these days.

>

> Yes, test-mingw has it.


Ahh cool :-)

--
Alex Bennée
diff mbox

Patch

diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index 19d4cc7..ee47085 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -101,31 +101,73 @@  docker:
 	@echo '    NOCACHE=1            Ignore cache when build images.'
 	@echo '    EXECUTABLE=<path>    Include executable in image.'
 
-docker-run-%: CMD = $(shell echo '$@' | sed -e 's/docker-run-\([^@]*\)@\(.*\)/\1/')
-docker-run-%: IMAGE = $(shell echo '$@' | sed -e 's/docker-run-\([^@]*\)@\(.*\)/\2/')
-docker-run-%: docker-qemu-src
+# This rule if for directly running against an arbitary docker target.
+# It is called by the expanded docker targets (e.g. make
+# docker-test-foo@bar) which will do additional verification.
+#
+# For example: make docker-run TEST="test-quick" IMAGE="debian:arm64" EXECUTABLE=./aarch64-linux-user/qemu-aarch64
+#
+docker-run: docker-qemu-src
 	@mkdir -p "$(DOCKER_CCACHE_DIR)"
-	@if test -z "$(IMAGE)" || test -z "$(CMD)"; \
-		then echo "Invalid target"; exit 1; \
+	@if test -z "$(IMAGE)" || test -z "$(TEST)"; \
+		then echo "Invalid target $(IMAGE)/$(TEST)"; exit 1; \
 	fi
-	$(if $(filter $(TESTS),$(CMD)),$(if $(filter $(IMAGES),$(IMAGE)), \
-		$(call quiet-command,\
-			if $(SRC_PATH)/tests/docker/docker.py images | \
-				awk '$$1=="qemu" && $$2=="$(IMAGE)"{found=1} END{exit(!found)}'; then \
-				$(SRC_PATH)/tests/docker/docker.py run $(if $V,,--rm) \
-				-t \
-				$(if $(DEBUG),-i,--net=none) \
-				-e TARGET_LIST=$(TARGET_LIST) \
+	$(if $(EXECUTABLE),						\
+		$(call quiet-command,					\
+			$(SRC_PATH)/tests/docker/docker.py update 	\
+			$(IMAGE) $(EXECUTABLE)))
+	$(call quiet-command,						\
+		$(SRC_PATH)/tests/docker/docker.py run 			\
+				-t 					\
+				$(if $V,,--rm) 				\
+				$(if $(DEBUG),-i,--net=none) 		\
+				-e TARGET_LIST=$(TARGET_LIST) 		\
 				-e EXTRA_CONFIGURE_OPTS=$(EXTRA_CONFIGURE_OPTS) \
-				-e V=$V -e J=$J -e DEBUG=$(DEBUG)\
-				-e CCACHE_DIR=/var/tmp/ccache \
+				-e V=$V -e J=$J -e DEBUG=$(DEBUG)	\
+				-e CCACHE_DIR=/var/tmp/ccache 		\
 				-v $$(readlink -e $(DOCKER_SRC_COPY)):/var/tmp/qemu:z$(COMMA)ro \
 				-v $(DOCKER_CCACHE_DIR):/var/tmp/ccache:z \
-				qemu:$(IMAGE) \
-				/var/tmp/qemu/run \
-				$(CMD); \
-			fi \
-			, "  RUN $(CMD) in $(IMAGE)")))
+				$(IMAGE) 				\
+				/var/tmp/qemu/run 			\
+				$(TEST), "  RUN $(TEST) in ${IMAGE}")
+
+#
+# Verification targets
+#
+# These targets help verify the test (CMD) and docker tag (IMAGE) are
+# part of the built in set of tests and images. You can still call the
+# docker-run target directly for testsing against arbitary images.
+#
+
+docker-verify-image-%: IMAGE = $(shell echo '$@' | sed -e 's/docker-verify-image-\([^@]*\)@\(.*\)/\2/')
+docker-verify-image-%:
+	@if test -z "$(IMAGE)";			 	\
+		then echo "Invalid image"; exit 1; 	\
+	fi
+	$(if $(findstring $(IMAGE), $(DOCKER_IMAGES))	\
+		, @echo "Verified $(IMAGE)"		\
+		, @echo "$(IMAGE) is not a known image"; exit 1)
+
+docker-verify-test-%: CMD = $(shell echo '$@' | sed -e 's/docker-verify-test-\([^@]*\)@\(.*\)/\1/')
+docker-verify-test-%:
+	@if test -z "$(CMD)"; 				\
+		then echo "Invalid test"; exit 1; 	\
+	fi
+	$(if $(findstring $(CMD), $(DOCKER_TESTS))	\
+		, @echo "Verified $(CMD)"		\
+		, @echo "$(CMD) is not a known test"; exit 1)
+
+# Run targets
+#
+# This will take a target such as docker-test-foo@bar and verify that:
+#  - the test test-foo is a known test
+#  - the image bar is a known image
+#
+# It will then call the docker-run
+docker-run-%: CMD = $(shell echo '$@' | sed -e 's/docker-run-\([^@]*\)@\(.*\)/\1/')
+docker-run-%: IMAGE = $(shell echo '$@' | sed -e 's/docker-run-\([^@]*\)@\(.*\)/\2/')
+docker-run-%: docker-verify-image-% docker-verify-test-%
+	@make docker-run TEST=$(CMD) IMAGE=qemu:$(IMAGE)
 
 docker-clean:
 	$(call quiet-command, $(SRC_PATH)/tests/docker/docker.py clean)