| Message ID | 20230224180857.1050220-1-alex.bennee@linaro.org |
|---|---|
| Headers | show |
| Series | testing/next: docker.py removal and kaniko updates | expand |
On 24/2/23 19:08, Alex Bennée wrote: > This series attempts to remove our dependence on the docker.py script > and build things directly with the appropriate tool. I've been > noodling around with how we build images on gitlab to see if they can > cache better because the normal case should be we don't need to > rebuild everything if the upstream distro hasn't updated its package > list. > > Anyway what do people think? Removing dind limitation is interesting. Unrelated, can we tag registry.gitlab.com/qemu-project's docker images along with releases?
On Tue, Feb 28, 2023 at 12:58:54PM +0100, Philippe Mathieu-Daudé wrote: > On 24/2/23 19:08, Alex Bennée wrote: > > This series attempts to remove our dependence on the docker.py script > > and build things directly with the appropriate tool. I've been > > noodling around with how we build images on gitlab to see if they can > > cache better because the normal case should be we don't need to > > rebuild everything if the upstream distro hasn't updated its package > > list. > > > > Anyway what do people think? > > Removing dind limitation is interesting. > > Unrelated, can we tag registry.gitlab.com/qemu-project's > docker images along with releases? Can you elaborate on this ? We're only using the images for CI purposes and they must always reflect the current state of git master. We're using a fixed docker tag 'latest', as that avoids the container registry growing arbitrarily large. Our CI rules should prevent the pipelines running on stable branches, so we shouldn't need container tags for stable. With regards, Daniel
On 28/2/23 13:58, Daniel P. Berrangé wrote: > On Tue, Feb 28, 2023 at 12:58:54PM +0100, Philippe Mathieu-Daudé wrote: >> On 24/2/23 19:08, Alex Bennée wrote: >>> This series attempts to remove our dependence on the docker.py script >>> and build things directly with the appropriate tool. I've been >>> noodling around with how we build images on gitlab to see if they can >>> cache better because the normal case should be we don't need to >>> rebuild everything if the upstream distro hasn't updated its package >>> list. >>> >>> Anyway what do people think? >> >> Removing dind limitation is interesting. >> >> Unrelated, can we tag registry.gitlab.com/qemu-project's >> docker images along with releases? > > Can you elaborate on this ? > > We're only using the images for CI purposes and they must always reflect > the current state of git master. We're using a fixed docker tag 'latest', > as that avoids the container registry growing arbitrarily large. > > Our CI rules should prevent the pipelines running on stable branches, > so we shouldn't need container tags for stable. I'm not suggesting keeping jobs to build, but doing a snapshot of the released containers. I.e. when we release 8.0, we should tag qemu/fedora:v8.0 and never touch it again. This is useful when bisecting pre-v8, but also to build pre-v8 and do performance comparison. One shouldn't have to upgrade such container (in particular when package mirror disappear), since it already contains all we need. Personally I'd like to keep generic and problematic ones: - qemu/fedora - qemu/fedora-win64-cross - qemu/debian-xtensa-cross
On Tue, Feb 28, 2023 at 02:29:12PM +0100, Philippe Mathieu-Daudé wrote: > On 28/2/23 13:58, Daniel P. Berrangé wrote: > > On Tue, Feb 28, 2023 at 12:58:54PM +0100, Philippe Mathieu-Daudé wrote: > > > On 24/2/23 19:08, Alex Bennée wrote: > > > > This series attempts to remove our dependence on the docker.py script > > > > and build things directly with the appropriate tool. I've been > > > > noodling around with how we build images on gitlab to see if they can > > > > cache better because the normal case should be we don't need to > > > > rebuild everything if the upstream distro hasn't updated its package > > > > list. > > > > > > > > Anyway what do people think? > > > > > > Removing dind limitation is interesting. > > > > > > Unrelated, can we tag registry.gitlab.com/qemu-project's > > > docker images along with releases? > > > > Can you elaborate on this ? > > > > We're only using the images for CI purposes and they must always reflect > > the current state of git master. We're using a fixed docker tag 'latest', > > as that avoids the container registry growing arbitrarily large. > > > > Our CI rules should prevent the pipelines running on stable branches, > > so we shouldn't need container tags for stable. > > I'm not suggesting keeping jobs to build, but doing a snapshot of the > released containers. > > I.e. when we release 8.0, we should tag qemu/fedora:v8.0 and never touch > it again. This is useful when bisecting pre-v8, but also to build pre-v8 > and do performance comparison. One shouldn't have to upgrade such > container (in particular when package mirror disappear), since it > already contains all we need. The main risk with this is the impact on our storage quota. With the OSS Program membership IIUC we get Ultimate level features which is 250 GB of storage, across git repos, pipeline cache/artifacts/logs, container registry. Currently they have no way to enforce that since their accounting of usage is not accurate enough. They're working on fixing that so at somepoint we'll be subject to the 250 GB limit. What I don't know is how much storage we're currently using across the /qemu-project namespace, and what extra is implied by taking a snapshot of our container registry 3 times a year. I'm expecting it to probably be in the high 10's of GB though for the container registry. With regards, Daniel
Daniel P. Berrangé <berrange@redhat.com> writes: > On Tue, Feb 28, 2023 at 02:29:12PM +0100, Philippe Mathieu-Daudé wrote: >> On 28/2/23 13:58, Daniel P. Berrangé wrote: >> > On Tue, Feb 28, 2023 at 12:58:54PM +0100, Philippe Mathieu-Daudé wrote: >> > > On 24/2/23 19:08, Alex Bennée wrote: >> > > > This series attempts to remove our dependence on the docker.py script >> > > > and build things directly with the appropriate tool. I've been >> > > > noodling around with how we build images on gitlab to see if they can >> > > > cache better because the normal case should be we don't need to >> > > > rebuild everything if the upstream distro hasn't updated its package >> > > > list. >> > > > >> > > > Anyway what do people think? >> > > >> > > Removing dind limitation is interesting. >> > > >> > > Unrelated, can we tag registry.gitlab.com/qemu-project's >> > > docker images along with releases? >> > >> > Can you elaborate on this ? >> > >> > We're only using the images for CI purposes and they must always reflect >> > the current state of git master. We're using a fixed docker tag 'latest', >> > as that avoids the container registry growing arbitrarily large. >> > >> > Our CI rules should prevent the pipelines running on stable branches, >> > so we shouldn't need container tags for stable. >> >> I'm not suggesting keeping jobs to build, but doing a snapshot of the >> released containers. >> >> I.e. when we release 8.0, we should tag qemu/fedora:v8.0 and never touch >> it again. This is useful when bisecting pre-v8, but also to build pre-v8 >> and do performance comparison. One shouldn't have to upgrade such >> container (in particular when package mirror disappear), since it >> already contains all we need. > > The main risk with this is the impact on our storage quota. With the > OSS Program membership IIUC we get Ultimate level features which > is 250 GB of storage, across git repos, pipeline cache/artifacts/logs, > container registry. > > Currently they have no way to enforce that since their accounting of > usage is not accurate enough. They're working on fixing that so at > somepoint we'll be subject to the 250 GB limit. > > What I don't know is how much storage we're currently using across > the /qemu-project namespace, and what extra is implied by taking > a snapshot of our container registry 3 times a year. I'm expecting > it to probably be in the high 10's of GB though for the container > registry. Currently we are using: 86.6 Gb of artefacts 28.5 Gb of containers 220 Mb of file uploads 24.8Mb of git repo We could probably cut down a lot of usage of artefacts by either not building full fat ones with debug symbols and passing between layers or tweaking the build system to prevent re-building of object files if the final binary is present in the file system.
On 28/2/23 14:57, Alex Bennée wrote: > > Daniel P. Berrangé <berrange@redhat.com> writes: > >> On Tue, Feb 28, 2023 at 02:29:12PM +0100, Philippe Mathieu-Daudé wrote: >>> On 28/2/23 13:58, Daniel P. Berrangé wrote: >>>> On Tue, Feb 28, 2023 at 12:58:54PM +0100, Philippe Mathieu-Daudé wrote: >>>>> On 24/2/23 19:08, Alex Bennée wrote: >>>>>> This series attempts to remove our dependence on the docker.py script >>>>>> and build things directly with the appropriate tool. I've been >>>>>> noodling around with how we build images on gitlab to see if they can >>>>>> cache better because the normal case should be we don't need to >>>>>> rebuild everything if the upstream distro hasn't updated its package >>>>>> list. >>>>>> >>>>>> Anyway what do people think? >>>>> >>>>> Removing dind limitation is interesting. >>>>> >>>>> Unrelated, can we tag registry.gitlab.com/qemu-project's >>>>> docker images along with releases? >>>> >>>> Can you elaborate on this ? >>>> >>>> We're only using the images for CI purposes and they must always reflect >>>> the current state of git master. We're using a fixed docker tag 'latest', >>>> as that avoids the container registry growing arbitrarily large. >>>> >>>> Our CI rules should prevent the pipelines running on stable branches, >>>> so we shouldn't need container tags for stable. >>> >>> I'm not suggesting keeping jobs to build, but doing a snapshot of the >>> released containers. >>> >>> I.e. when we release 8.0, we should tag qemu/fedora:v8.0 and never touch >>> it again. This is useful when bisecting pre-v8, but also to build pre-v8 >>> and do performance comparison. One shouldn't have to upgrade such >>> container (in particular when package mirror disappear), since it >>> already contains all we need. >> >> The main risk with this is the impact on our storage quota. With the >> OSS Program membership IIUC we get Ultimate level features which >> is 250 GB of storage, across git repos, pipeline cache/artifacts/logs, >> container registry. >> >> Currently they have no way to enforce that since their accounting of >> usage is not accurate enough. They're working on fixing that so at >> somepoint we'll be subject to the 250 GB limit. >> >> What I don't know is how much storage we're currently using across >> the /qemu-project namespace, and what extra is implied by taking >> a snapshot of our container registry 3 times a year. I'm expecting >> it to probably be in the high 10's of GB though for the container >> registry. Maybe we can keep (a subset of) the stable release container images and store them on a registry outside of gitlab. That would be nice, but if I'm the only one interested and nobody felt this was useful before, no need to spend more energy on this topic. > Currently we are using: > > 86.6 Gb of artefacts ^ transient > 28.5 Gb of containers ^ constant > 220 Mb of file uploads > 24.8Mb of git repo > > We could probably cut down a lot of usage of artefacts by either not > building full fat ones with debug symbols and passing between layers or > tweaking the build system to prevent re-building of object files if the > final binary is present in the file system. >