diff mbox series

[v2,29/30] tests/docker: remove FROM qemu/ support from docker.py

Message ID 20220914155950.804707-30-alex.bennee@linaro.org
State Superseded
Headers show
Series testing/next pre-PR (testing update and mips deprecation) | expand

Commit Message

Alex Bennée Sept. 14, 2022, 3:59 p.m. UTC 
We want to migrate from docker.py to building our images directly with
docker/podman. Before we get there we need to make sure we don't
re-introduce our layered builds so bug out if we see FROM qemu/ in a
Dockerfile.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Acked-by: Thomas Huth <thuth@redhat.com>
Message-Id: <20220826172128.353798-24-alex.bennee@linaro.org>
---
 tests/docker/docker.py | 38 ++++++++++----------------------------
 1 file changed, 10 insertions(+), 28 deletions(-)

Comments

Richard Henderson Sept. 15, 2022, 8:41 a.m. UTC | #1 
On 9/14/22 16:59, Alex Bennée wrote:
> We want to migrate from docker.py to building our images directly with
> docker/podman. Before we get there we need to make sure we don't
> re-introduce our layered builds so bug out if we see FROM qemu/ in a
> Dockerfile.
> 
> Signed-off-by: Alex Bennée<alex.bennee@linaro.org>
> Acked-by: Thomas Huth<thuth@redhat.com>
> Message-Id:<20220826172128.353798-24-alex.bennee@linaro.org>
> ---
>   tests/docker/docker.py | 38 ++++++++++----------------------------
>   1 file changed, 10 insertions(+), 28 deletions(-)

Reviewed-by: Richard Henderson <richard.henderson@linaro.org>

r~
Stefan Weil Oct. 29, 2022, 9:38 a.m. UTC | #2 
Am 15.09.22 um 10:41 schrieb Richard Henderson:
> On 9/14/22 16:59, Alex Bennée wrote:
>> We want to migrate from docker.py to building our images directly with
>> docker/podman. Before we get there we need to make sure we don't
>> re-introduce our layered builds so bug out if we see FROM qemu/ in a
>> Dockerfile.
>>
>> Signed-off-by: Alex Bennée<alex.bennee@linaro.org>
>> Acked-by: Thomas Huth<thuth@redhat.com>
>> Message-Id:<20220826172128.353798-24-alex.bennee@linaro.org>
>> ---
>>   tests/docker/docker.py | 38 ++++++++++----------------------------
>>   1 file changed, 10 insertions(+), 28 deletions(-)
> 
> Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
> 
> r~
> 

I am not sure whether that is related, but with latest QEMU code I now 
get an e-mail "SECURITY information [...]" each time when I run the 
configure script (or just `tests/docker/docker.py probe`):

qemu.weilnetz.de : Oct 29 09:32:32 : stefan : a password is required ; 
TTY=pts/0 ; PWD=/home/stefan/qemu/bin/ndebug/i686-w64-mingw32 ; 
USER=root ; COMMAND=docker version

There is no docker installation on my Debian GNU Linux build machine.

Stefan
Paolo Bonzini Oct. 29, 2022, 7:46 p.m. UTC | #3 
Il sab 29 ott 2022, 11:38 Stefan Weil <sw@weilnetz.de> ha scritto:

> Am 15.09.22 um 10:41 schrieb Richard Henderson:
> > On 9/14/22 16:59, Alex Bennée wrote:
> >> We want to migrate from docker.py to building our images directly with
> >> docker/podman. Before we get there we need to make sure we don't
> >> re-introduce our layered builds so bug out if we see FROM qemu/ in a
> >> Dockerfile.
> >>
> >> Signed-off-by: Alex Bennée<alex.bennee@linaro.org>
> >> Acked-by: Thomas Huth<thuth@redhat.com>
> >> Message-Id:<20220826172128.353798-24-alex.bennee@linaro.org>
> >> ---
> >>   tests/docker/docker.py | 38 ++++++++++----------------------------
> >>   1 file changed, 10 insertions(+), 28 deletions(-)
> >
> > Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
> >
> > r~
> >
>
> I am not sure whether that is related, but with latest QEMU code I now
> get an e-mail "SECURITY information [...]" each time when I run the
> configure script (or just `tests/docker/docker.py probe`):
>
> qemu.weilnetz.de : Oct 29 09:32:32 : stefan : a password is required ;
> TTY=pts/0 ; PWD=/home/stefan/qemu/bin/ndebug/i686-w64-mingw32 ;
> USER=root ; COMMAND=docker version
>
> There is no docker installation on my Debian GNU Linux build machine.
>

It's not related, configure is checking whether "sudo docker" works.

Paolo





> Stefan
>
Stefan Weil Oct. 29, 2022, 9:13 p.m. UTC | #4 
Am 29.10.22 um 21:46 schrieb Paolo Bonzini:

> Il sab 29 ott 2022, 11:38 Stefan Weil <sw@weilnetz.de> ha scritto:
>
>     Am 15.09.22 um 10:41 schrieb Richard Henderson:
>     > On 9/14/22 16:59, Alex Bennée wrote:
>     >> We want to migrate from docker.py to building our images
>     directly with
>     >> docker/podman. Before we get there we need to make sure we don't
>     >> re-introduce our layered builds so bug out if we see FROM qemu/
>     in a
>     >> Dockerfile.
>     >>
>     >> Signed-off-by: Alex Bennée<alex.bennee@linaro.org>
>     >> Acked-by: Thomas Huth<thuth@redhat.com>
>     >> Message-Id:<20220826172128.353798-24-alex.bennee@linaro.org>
>     >> ---
>     >>   tests/docker/docker.py | 38
>     ++++++++++----------------------------
>     >>   1 file changed, 10 insertions(+), 28 deletions(-)
>     >
>     > Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
>     >
>     > r~
>     >
>
>     I am not sure whether that is related, but with latest QEMU code I
>     now
>     get an e-mail "SECURITY information [...]" each time when I run the
>     configure script (or just `tests/docker/docker.py probe`):
>
>     qemu.weilnetz.de <http://qemu.weilnetz.de> : Oct 29 09:32:32 :
>     stefan : a password is required ;
>     TTY=pts/0 ; PWD=/home/stefan/qemu/bin/ndebug/i686-w64-mingw32 ;
>     USER=root ; COMMAND=docker version
>
>     There is no docker installation on my Debian GNU Linux build machine.
>
>
> It's not related, configure is checking whether "sudo docker" works.


You are right. "git bisect" finds that the security e-mails occur since 
another commit.

Stefan


c4575b59155e2e0065c29d56496dbf3e6ffd035b is the first bad commit
commit c4575b59155e2e0065c29d56496dbf3e6ffd035b
Author: Paolo Bonzini <pbonzini@redhat.com>
Date:   Thu Sep 29 12:41:58 2022 +0100

     configure: store container engine in config-host.mak

     In preparation for removing $(DOCKER_SCRIPT) from the tests/tcg 
configuration
     files, have Make use the same container engine that had been probed at
     configure time.

     Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
     Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
     Message-Id: <20220929114231.583801-19-alex.bennee@linaro.org>

  configure                     | 11 ++++++++---
  tests/docker/Makefile.include |  2 +-
  2 files changed, 9 insertions(+), 4 deletions(-)
Paolo Bonzini Oct. 30, 2022, 2:17 p.m. UTC | #5 
I think sudo  needs to be changed to "sudo -n" in docker.py. Thanks for the
report!

Paolo

Il sab 29 ott 2022, 23:13 Stefan Weil <sw@weilnetz.de> ha scritto:

> Am 29.10.22 um 21:46 schrieb Paolo Bonzini:
>
> Il sab 29 ott 2022, 11:38 Stefan Weil <sw@weilnetz.de> ha scritto:
>
>> Am 15.09.22 um 10:41 schrieb Richard Henderson:
>> > On 9/14/22 16:59, Alex Bennée wrote:
>> >> We want to migrate from docker.py to building our images directly with
>> >> docker/podman. Before we get there we need to make sure we don't
>> >> re-introduce our layered builds so bug out if we see FROM qemu/ in a
>> >> Dockerfile.
>> >>
>> >> Signed-off-by: Alex Bennée<alex.bennee@linaro.org>
>> >> Acked-by: Thomas Huth<thuth@redhat.com>
>> >> Message-Id:<20220826172128.353798-24-alex.bennee@linaro.org>
>> >> ---
>> >>   tests/docker/docker.py | 38 ++++++++++----------------------------
>> >>   1 file changed, 10 insertions(+), 28 deletions(-)
>> >
>> > Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
>> >
>> > r~
>> >
>>
>> I am not sure whether that is related, but with latest QEMU code I now
>> get an e-mail "SECURITY information [...]" each time when I run the
>> configure script (or just `tests/docker/docker.py probe`):
>>
>> qemu.weilnetz.de : Oct 29 09:32:32 : stefan : a password is required ;
>> TTY=pts/0 ; PWD=/home/stefan/qemu/bin/ndebug/i686-w64-mingw32 ;
>> USER=root ; COMMAND=docker version
>>
>> There is no docker installation on my Debian GNU Linux build machine.
>>
>
> It's not related, configure is checking whether "sudo docker" works.
>
>
> You are right. "git bisect" finds that the security e-mails occur since
> another commit.
>
> Stefan
>
>
> c4575b59155e2e0065c29d56496dbf3e6ffd035b is the first bad commit
> commit c4575b59155e2e0065c29d56496dbf3e6ffd035b
> Author: Paolo Bonzini <pbonzini@redhat.com> <pbonzini@redhat.com>
> Date:   Thu Sep 29 12:41:58 2022 +0100
>
>     configure: store container engine in config-host.mak
>
>     In preparation for removing $(DOCKER_SCRIPT) from the tests/tcg
> configuration
>     files, have Make use the same container engine that had been probed at
>     configure time.
>
>     Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
> <pbonzini@redhat.com>
>     Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> <alex.bennee@linaro.org>
>     Message-Id: <20220929114231.583801-19-alex.bennee@linaro.org>
> <20220929114231.583801-19-alex.bennee@linaro.org>
>
>  configure                     | 11 ++++++++---
>  tests/docker/Makefile.include |  2 +-
>  2 files changed, 9 insertions(+), 4 deletions(-)
>
>
diff mbox series

Patch

diff --git a/tests/docker/docker.py b/tests/docker/docker.py
index d0af2861b8..3a1ed7cb18 100755
--- a/tests/docker/docker.py
+++ b/tests/docker/docker.py
@@ -205,22 +205,17 @@  def _read_qemu_dockerfile(img_name):
     return _read_dockerfile(df)
 
 
-def _dockerfile_preprocess(df):
-    out = ""
+def _dockerfile_verify_flat(df):
+    "Verify we do not include other qemu/ layers"
     for l in df.splitlines():
         if len(l.strip()) == 0 or l.startswith("#"):
             continue
         from_pref = "FROM qemu/"
         if l.startswith(from_pref):
-            # TODO: Alternatively we could replace this line with "FROM $ID"
-            # where $ID is the image's hex id obtained with
-            #    $ docker images $IMAGE --format="{{.Id}}"
-            # but unfortunately that's not supported by RHEL 7.
-            inlining = _read_qemu_dockerfile(l[len(from_pref):])
-            out += _dockerfile_preprocess(inlining)
-            continue
-        out += l + "\n"
-    return out
+            print("We no longer support multiple QEMU layers.")
+            print("Dockerfiles should be flat, ideally created by lcitool")
+            return False
+    return True
 
 
 class Docker(object):
@@ -309,23 +304,10 @@  def build_image(self, tag, docker_dir, dockerfile,
         if argv is None:
             argv = []
 
-        # pre-calculate the docker checksum before any
-        # substitutions we make for caching
-        checksum = _text_checksum(_dockerfile_preprocess(dockerfile))
+        if not _dockerfile_verify_flat(dockerfile):
+            return -1
 
-        if registry is not None:
-            sources = re.findall("FROM qemu\/(.*)", dockerfile)
-            # Fetch any cache layers we can, may fail
-            for s in sources:
-                pull_args = ["pull", "%s/qemu/%s" % (registry, s)]
-                if self._do(pull_args, quiet=quiet) != 0:
-                    registry = None
-                    break
-            # Make substitutions
-            if registry is not None:
-                dockerfile = dockerfile.replace("FROM qemu/",
-                                                "FROM %s/qemu/" %
-                                                (registry))
+        checksum = _text_checksum(dockerfile)
 
         tmp_df = tempfile.NamedTemporaryFile(mode="w+t",
                                              encoding='utf-8',
@@ -371,7 +353,7 @@  def image_matches_dockerfile(self, tag, dockerfile):
             checksum = self.get_image_dockerfile_checksum(tag)
         except Exception:
             return False
-        return checksum == _text_checksum(_dockerfile_preprocess(dockerfile))
+        return checksum == _text_checksum(dockerfile)
 
     def run(self, cmd, keep, quiet, as_user=False):
         label = uuid.uuid4().hex