| Message ID | 20201007035154.109-3-luoyonggang@gmail.com |
|---|---|
| State | New |
| Headers | show |
| Series | Improve cirrus msys2 | expand |
On 07/10/20 05:51, Yonggang Luo wrote: > Using a prepared https://github.com/lygstate/qemu/releases/download/v5.1.0/msys2-x86_64.tar.xz > for speed up the msys2/mingw prepare speed on the cirrus CI. > This file are generated by the msys2.docker The speedup is nice, but we need to be able to regenerate the tarball without access to a Windows machine. Otherwise whoever adds a new dependency cannot test it himself with CI. Ideally, the tarball would be generated by the containers job in the Gitlab CI, but I'm not sure if that's possible. On top of this, whoever commits this needs to replace the URL with something under download.qemu.org, but that's the easy part. Paolo
On Wed, Oct 07, 2020 at 10:18:48AM +0200, Paolo Bonzini wrote: > On 07/10/20 05:51, Yonggang Luo wrote: > > Using a prepared https://github.com/lygstate/qemu/releases/download/v5.1.0/msys2-x86_64.tar.xz > > for speed up the msys2/mingw prepare speed on the cirrus CI. > > This file are generated by the msys2.docker > > The speedup is nice, but we need to be able to regenerate the tarball > without access to a Windows machine. Otherwise whoever adds a new > dependency cannot test it himself with CI. > > Ideally, the tarball would be generated by the containers job in the > Gitlab CI, but I'm not sure if that's possible. IMHO any caching should be integrated into the Cirrus CI job itself. Cirrus has builtin support for caching the contents of a folder, so the first time the job runs it should populate the cache, and that can be used thereafter. Regards, Daniel
On 07/10/2020 10.18, Paolo Bonzini wrote: [...] > > On top of this, whoever commits this needs to replace the URL with > something under download.qemu.org, but that's the easy part. I agree with Daniel - please let's not get go this way again. We've had this with the various Free-/Net-/OpenBSD images in the past, and then we were suddenly stuck with a fixed image where most people were not able to change it anymore. That was very frustrating. Apart from that, you also have to respect the GPL if you provide binaries ... so you'd need to gather all the sources for that blob, too, and be ready to provide those in case somebody asks... Yonggang, could you please try to use the caching mechanism of Cirrus-CI instead? See https://cirrus-ci.org/guide/writing-tasks/#cache-instruction ... then we're a little bit more independent from downloading the msys2 for each CI run. Thomas
On Wed, Oct 7, 2020 at 4:41 PM Thomas Huth <thuth@redhat.com> wrote: > > On 07/10/2020 10.18, Paolo Bonzini wrote: > [...] > > > > On top of this, whoever commits this needs to replace the URL with > > something under download.qemu.org, but that's the easy part. > > I agree with Daniel - please let's not get go this way again. We've had this > with the various Free-/Net-/OpenBSD images in the past, and then we were > suddenly stuck with a fixed image where most people were not able to change > it anymore. That was very frustrating. > > Apart from that, you also have to respect the GPL if you provide binaries > ... so you'd need to gather all the sources for that blob, too, and be ready > to provide those in case somebody asks... > > Yonggang, could you please try to use the caching mechanism of Cirrus-CI > instead? See https://cirrus-ci.org/guide/writing-tasks/#cache-instruction > ... then we're a little bit more independent from downloading the msys2 for > each CI run. Looks good for me, let's me investigate > > Thomas > -- 此致 礼 罗勇刚 Yours sincerely, Yonggang Luo <div dir="ltr"><br><br>On Wed, Oct 7, 2020 at 4:41 PM Thomas Huth <<a href="mailto:thuth@redhat.com">thuth@redhat.com</a>> wrote:<br>><br>> On 07/10/2020 10.18, Paolo Bonzini wrote:<br>> [...]<br>> ><br>> > On top of this, whoever commits this needs to replace the URL with<br>> > something under <a href="http://download.qemu.org">download.qemu.org</a>, but that's the easy part.<br>><br>> I agree with Daniel - please let's not get go this way again. We've had this<br>> with the various Free-/Net-/OpenBSD images in the past, and then we were<br>> suddenly stuck with a fixed image where most people were not able to change<br>> it anymore. That was very frustrating.<br>><br>> Apart from that, you also have to respect the GPL if you provide binaries<br>> ... so you'd need to gather all the sources for that blob, too, and be ready<br>> to provide those in case somebody asks...<br>><br>> Yonggang, could you please try to use the caching mechanism of Cirrus-CI<br>> instead? See <a href="https://cirrus-ci.org/guide/writing-tasks/#cache-instruction">https://cirrus-ci.org/guide/writing-tasks/#cache-instruction</a><br>> ... then we're a little bit more independent from downloading the msys2 for<br>> each CI run.<div>Looks good for me, let's me investigate<br>><br>> Thomas<br>><br><br><br>--<br> 此致<br>礼<br>罗勇刚<br>Yours<br> sincerely,<br>Yonggang Luo</div></div>
diff --git a/.cirrus.yml b/.cirrus.yml index d58782ce67..bf5a0990ec 100644 --- a/.cirrus.yml +++ b/.cirrus.yml @@ -47,63 +47,33 @@ macos_xcode_task: windows_msys2_task: windows_container: - image: cirrusci/windowsservercore:cmake + image: cirrusci/windowsservercore:2019 os_version: 2019 cpu: 8 memory: 8G env: + CIRRUS_SHELL: powershell MSYS: winsymlinks:nativestrict MSYSTEM: MINGW64 CHERE_INVOKING: 1 - printenv_script: - - C:\tools\msys64\usr\bin\bash.exe -lc 'printenv' install_script: - - C:\tools\msys64\usr\bin\bash.exe -lc "cd /c/tools && - curl -O http://repo.msys2.org/msys/x86_64/msys2-keyring-r21.b39fb11-1-any.pkg.tar.xz" - - C:\tools\msys64\usr\bin\bash.exe -lc "cd /c/tools && - curl -O http://repo.msys2.org/msys/x86_64/msys2-keyring-r21.b39fb11-1-any.pkg.tar.xz.sig" - - C:\tools\msys64\usr\bin\bash.exe -lc "cd /c/tools && - pacman -U --noconfirm msys2-keyring-r21.b39fb11-1-any.pkg.tar.xz" - - C:\tools\msys64\usr\bin\bash.exe -lc "pacman --noconfirm -Sy" - - C:\tools\msys64\usr\bin\bash.exe -lc "pacman --noconfirm -S --needed - bash pacman pacman-mirrors msys2-runtime" - - taskkill /F /IM gpg-agent.exe - - C:\tools\msys64\usr\bin\bash.exe -lc "pacman --noconfirm -Su" - - C:\tools\msys64\usr\bin\bash.exe -lc "pacman --noconfirm -S --needed - base-devel - git - mingw-w64-x86_64-python - mingw-w64-x86_64-python-setuptools - mingw-w64-x86_64-toolchain - mingw-w64-x86_64-capstone - mingw-w64-x86_64-SDL2 - mingw-w64-x86_64-SDL2_image - mingw-w64-x86_64-gtk3 - mingw-w64-x86_64-glib2 - mingw-w64-x86_64-ninja - mingw-w64-x86_64-make - mingw-w64-x86_64-lzo2 - mingw-w64-x86_64-zstd - mingw-w64-x86_64-libjpeg-turbo - mingw-w64-x86_64-pixman - mingw-w64-x86_64-libgcrypt - mingw-w64-x86_64-libpng - mingw-w64-x86_64-libssh - mingw-w64-x86_64-libxml2 - mingw-w64-x86_64-snappy - mingw-w64-x86_64-libusb - mingw-w64-x86_64-usbredir - mingw-w64-x86_64-libtasn1 - mingw-w64-x86_64-nettle - mingw-w64-x86_64-cyrus-sasl - mingw-w64-x86_64-curl - mingw-w64-x86_64-gnutls - mingw-w64-x86_64-zstd" + - | + Write-Output $env:PATH + md C:\tools + $start_time = Get-Date + bitsadmin /transfer msys_download /dynamic /download /priority FOREGROUND https://github.com/lygstate/qemu/releases/download/v5.1.0/msys2-x86_64.tar.xz C:\tools\msys2-x86_64.tar.xz + Write-Output "Download time taken: $((Get-Date).Subtract($start_time).Seconds) second(s)" + $start_time = Get-Date + cd C:\tools + choco install -y --no-progress 7zip + cmd /C "7z x msys2-x86_64.tar.xz -so | 7z x -aoa -simsys2-x86_64.tar -ttar -omsys64" + Write-Output "Extract time taken: $((Get-Date).Subtract($start_time).Seconds) second(s)" + script: - C:\tools\msys64\usr\bin\bash.exe -lc "mkdir build" - C:\tools\msys64\usr\bin\bash.exe -lc "cd build && ../configure --python=python3 --ninja=ninja --target-list-exclude=i386-softmmu,arm-softmmu,ppc-softmmu,mips-softmmu" - - C:\tools\msys64\usr\bin\bash.exe -lc "cd build && make -j$NUMBER_OF_PROCESSORS" + - C:\tools\msys64\usr\bin\bash.exe -lc "cd build && make -j8" test_script: - C:\tools\msys64\usr\bin\bash.exe -lc "cd build && make V=1 check"
Using a prepared https://github.com/lygstate/qemu/releases/download/v5.1.0/msys2-x86_64.tar.xz for speed up the msys2/mingw prepare speed on the cirrus CI. This file are generated by the msys2.docker Signed-off-by: Yonggang Luo <luoyonggang@gmail.com> --- .cirrus.yml | 60 ++++++++++++++--------------------------------------- 1 file changed, 15 insertions(+), 45 deletions(-)