diff mbox series

[7/7] tests/gitlab: use kaniko to build images

Message ID 20230224180857.1050220-8-alex.bennee@linaro.org
State New
Headers show
Series testing/next: docker.py removal and kaniko updates | expand

Commit Message

Alex Bennée Feb. 24, 2023, 6:08 p.m. UTC
Apparently the docker-in-docker approach has some flaws including
needing privileged mode to run and being quite slow. An alternative
approach is to use Google's kaniko tool. It also works across
different gitlab executors.

Following the gitlab example code we drop all the direct docker calls
and usage of the script and make a direct call to kaniko and hope the
images are cacheable by others.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
 .gitlab-ci.d/container-template.yml | 23 +++++++++--------------
 1 file changed, 9 insertions(+), 14 deletions(-)

Comments

Daniel P. Berrangé Feb. 28, 2023, 10:53 a.m. UTC | #1
On Fri, Feb 24, 2023 at 06:08:57PM +0000, Alex Bennée wrote:
> Apparently the docker-in-docker approach has some flaws including
> needing privileged mode to run and being quite slow. An alternative
> approach is to use Google's kaniko tool. It also works across
> different gitlab executors.

Interesting, I've not come across this tool before.

> Following the gitlab example code we drop all the direct docker calls
> and usage of the script and make a direct call to kaniko and hope the
> images are cacheable by others.



> 
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> ---
>  .gitlab-ci.d/container-template.yml | 23 +++++++++--------------
>  1 file changed, 9 insertions(+), 14 deletions(-)
> 
> diff --git a/.gitlab-ci.d/container-template.yml b/.gitlab-ci.d/container-template.yml
> index c434b9c8f3..c2d7950df8 100644
> --- a/.gitlab-ci.d/container-template.yml
> +++ b/.gitlab-ci.d/container-template.yml
> @@ -1,22 +1,17 @@
>  .container_job_template:
>    extends: .base_job_template
> -  image: docker:stable
> +  image:
> +    name: gcr.io/kaniko-project/executor:v1.9.0-debug
> +    entrypoint: [""]
>    stage: containers
> -  services:
> -    - docker:dind
>    before_script:
>      - export TAG="$CI_REGISTRY_IMAGE/qemu/$NAME:latest"
> -    - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/$NAME:latest"
> -    - apk add python3
> -    - docker info
> -    - docker login $CI_REGISTRY -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD"
> +    - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/qemu/$NAME:latest"
>    script:
>      - echo "TAG:$TAG"
>      - echo "COMMON_TAG:$COMMON_TAG"
> -    - ./tests/docker/docker.py --engine docker build
> -          -t "qemu/$NAME" -f "tests/docker/dockerfiles/$NAME.docker"
> -          -r $CI_REGISTRY/qemu-project/qemu
> -    - docker tag "qemu/$NAME" "$TAG"
> -    - docker push "$TAG"
> -  after_script:
> -    - docker logout
> +    - /kaniko/executor
> +          --reproducible
> +          --context "${CI_PROJECT_DIR}"
> +          --dockerfile "${CI_PROJECT_DIR}/tests/docker/dockerfiles/$NAME.docker"
> +          --destination "${TAG}"

AFAICT from reading the docs, this does not allow for caching of layers
from the previous built image, so will always do a from scratch build.
This feels like it would kill any performance benefits over docker.

https://github.com/GoogleContainerTools/kaniko#caching

  "Users can opt into caching by setting the --cache=true flag. A
   remote repository for storing cached layers can be provided via
   the --cache-repo flag. If this flag isn't provided, a cached
   repo will be inferred from the --destination provided."

So it would sound like we need --cache=true adding to this command,
and in theory it should prime its cache from ${TAG}, but i'm not
entirely confident in that without testing. Also does not seem
like its psosible to have the dual cache COMMON_TAG + TAG, which
is particularly important for forks, since COMMON_TAG is more
likely to be up2date.

We could live with the latter restriction if we *always* used
COMMON_TAG as the cache, because 99% of the time forks are
going to just be using dockerfiles that match upstream. Only
those few contributors who have a branch that's changing
the dockerfiles would benefit from pointing the cache at
their fork.

So we could achieve that I believe using

     - /kaniko/executor
           --reproducible
           --context "${CI_PROJECT_DIR}"
	   --cache=true
	   --cache-repo "${COMMON_TAG}"
           --dockerfile "${CI_PROJECT_DIR}/tests/docker/dockerfiles/$NAME.docker"
           --destination "${TAG}"

With regards,
Daniel
diff mbox series

Patch

diff --git a/.gitlab-ci.d/container-template.yml b/.gitlab-ci.d/container-template.yml
index c434b9c8f3..c2d7950df8 100644
--- a/.gitlab-ci.d/container-template.yml
+++ b/.gitlab-ci.d/container-template.yml
@@ -1,22 +1,17 @@ 
 .container_job_template:
   extends: .base_job_template
-  image: docker:stable
+  image:
+    name: gcr.io/kaniko-project/executor:v1.9.0-debug
+    entrypoint: [""]
   stage: containers
-  services:
-    - docker:dind
   before_script:
     - export TAG="$CI_REGISTRY_IMAGE/qemu/$NAME:latest"
-    - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/$NAME:latest"
-    - apk add python3
-    - docker info
-    - docker login $CI_REGISTRY -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD"
+    - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/qemu/$NAME:latest"
   script:
     - echo "TAG:$TAG"
     - echo "COMMON_TAG:$COMMON_TAG"
-    - ./tests/docker/docker.py --engine docker build
-          -t "qemu/$NAME" -f "tests/docker/dockerfiles/$NAME.docker"
-          -r $CI_REGISTRY/qemu-project/qemu
-    - docker tag "qemu/$NAME" "$TAG"
-    - docker push "$TAG"
-  after_script:
-    - docker logout
+    - /kaniko/executor
+          --reproducible
+          --context "${CI_PROJECT_DIR}"
+          --dockerfile "${CI_PROJECT_DIR}/tests/docker/dockerfiles/$NAME.docker"
+          --destination "${TAG}"