mbox

[PULL,00/35] Testing, docs, semihosting and plugin updates

Message ID 20230126112250.2584701-1-alex.bennee@linaro.org
State New
Headers show

Pull-request

https://gitlab.com/stsquad/qemu.git tags/pull-jan-omnibus-260123-1

Message

Alex Bennée Jan. 26, 2023, 11:22 a.m. UTC
The following changes since commit 13356edb87506c148b163b8c7eb0695647d00c2a:

  Merge tag 'block-pull-request' of https://gitlab.com/stefanha/qemu into staging (2023-01-24 09:45:33 +0000)

are available in the Git repository at:

  https://gitlab.com/stsquad/qemu.git tags/pull-jan-omnibus-260123-1

for you to fetch changes up to d200653282f50df2a994a0609f88baa6a5889f80:

  plugins: Iterate on cb_lists in qemu_plugin_user_exit (2023-01-26 11:12:01 +0000)

----------------------------------------------------------------
Testing, docs, semihosting and plugin updates

  - update playbooks for custom runners
  - add section timing support to gitlab
  - upgrade fedora images to 37
  - purge perl from the build system and deps
  - disable unstable tests in CI
  - improve intro, emulation and semihosting docs
  - semihosting bug fix and O_BINARY default
  - add memory-sve test
  - fix some races in qht
  - improve plugin handling of memory helpers
  - optimise plugin hooks
  - fix some plugin deadlocks

----------------------------------------------------------------
Alex Bennée (12):
      scripts/ci: update gitlab-runner playbook to use latest runner
      gitlab: just use plain --cc=clang for custom runner build
      tests/unit: drop hacky race avoidance in test-io-channel-command
      gitlab: add lsan suppression file to workaround tcmalloc issues
      tests/docker: drop debian-tricore-cross's partial status
      tests/tcg: skip the vma-pthread test on CI
      gitlab: wrap up test results for custom runners
      docs: add hotlinks to about preface text
      docs: add a new section to outline emulation support
      semihosting: add semihosting section to the docs
      docs: add an introduction to the system docs
      tests/tcg: add memory-sve test for aarch64

Emilio Cota (9):
      cpu: free cpu->tb_jmp_cache with RCU
      util/qht: add missing atomic_set(hashes[i])
      thread: de-const qemu_spin_destroy
      util/qht: use striped locks under TSAN
      plugins: make qemu_plugin_user_exit's locking order consistent with fork_start's
      plugins: fix optimization in plugin_gen_disable_mem_helpers
      translator: always pair plugin_gen_insn_{start, end} calls
      tcg: exclude non-memory effecting helpers from instrumentation
      cpu-exec: assert that plugin_mem_cbs is NULL after execution

Evgeny Iakovlev (1):
      semihosting: add O_BINARY flag in host_open for NT compatibility

Keith Packard (1):
      semihosting: Write back semihosting data before completion callback

Marc-André Lureau (7):
      build-sys: fix crlf-ending C code
      .gitlab-ci.d/windows: do not disable opengl
      meson: replace Perl usage with Python
      docs: drop texinfo options
      Update lcitool and fedora to 37
      lcitool: drop perl from QEMU project/dependencies
      lcitool: drop texinfo from QEMU project/dependencies

Mark Cave-Ayland (1):
      gitlab: add FF_SCRIPT_SECTIONS for timings

Philippe Mathieu-Daudé (1):
      tests/docker: Install flex in debian-tricore-cross

Richard Henderson (2):
      tests/tcg: Use SIGKILL for timeout
      plugins: Iterate on cb_lists in qemu_plugin_user_exit

Thomas Huth (1):
      MAINTAINERS: Fix the entry for tests/tcg/nios2

 MAINTAINERS                                        |   2 +-
 docs/about/emulation.rst                           | 192 ++++++++++++++++++
 docs/about/index.rst                               |  17 +-
 docs/conf.py                                       |  13 --
 docs/devel/tcg-plugins.rst                         |   2 +
 docs/interop/live-block-operations.rst             |   2 +
 docs/interop/qemu-qmp-ref.rst                      |   2 +
 docs/system/arm/emulation.rst                      |   2 +
 docs/system/index.rst                              |   4 +-
 docs/system/introduction.rst                       | 220 +++++++++++++++++++++
 docs/system/multi-process.rst                      |   2 +
 docs/system/quickstart.rst                         |  21 --
 docs/tools/index.rst                               |   2 +
 docs/user/index.rst                                |   2 +
 meson.build                                        |   2 +-
 accel/tcg/plugin-helpers.h                         |   4 +-
 accel/tcg/tb-jmp-cache.h                           |   1 +
 include/qemu/plugin.h                              |  11 ++
 include/qemu/thread.h                              |   5 +-
 include/tcg/tcg.h                                  |   2 +
 accel/tcg/cpu-exec.c                               |   5 +-
 accel/tcg/plugin-gen.c                             |  26 ++-
 accel/tcg/translator.c                             |  15 +-
 cpu.c                                              |  11 +-
 plugins/core.c                                     |  23 ++-
 semihosting/syscalls.c                             |  28 +--
 tcg/tcg.c                                          |   6 +-
 tests/unit/test-io-channel-command.c               |  14 +-
 util/qht.c                                         |  97 +++++++--
 .gitlab-ci.d/base.yml                              |   5 +
 .gitlab-ci.d/buildtest.yml                         |   1 +
 .gitlab-ci.d/cirrus/freebsd-12.vars                |   2 +-
 .gitlab-ci.d/cirrus/freebsd-13.vars                |   2 +-
 .gitlab-ci.d/cirrus/macos-12.vars                  |   2 +-
 .gitlab-ci.d/custom-runners.yml                    |  11 ++
 .gitlab-ci.d/custom-runners/ubuntu-20.04-s390x.yml |  13 +-
 .../custom-runners/ubuntu-22.04-aarch32.yml        |   2 +-
 .../custom-runners/ubuntu-22.04-aarch64.yml        |  15 +-
 .gitlab-ci.d/windows.yml                           |   5 +-
 qemu-options.hx                                    |  30 ++-
 scripts/ci/setup/build-environment.yml             |   1 -
 scripts/ci/setup/gitlab-runner.yml                 |  56 ++----
 scripts/ci/setup/vars.yml.template                 |   2 -
 scripts/oss-fuzz/lsan_suppressions.txt             |   2 +
 scripts/shaderinclude.pl                           |  16 --
 scripts/shaderinclude.py                           |  26 +++
 tests/docker/Makefile.include                      |   1 -
 tests/docker/dockerfiles/alpine.docker             |   2 -
 tests/docker/dockerfiles/centos8.docker            |   2 -
 tests/docker/dockerfiles/debian-amd64-cross.docker |   4 +-
 tests/docker/dockerfiles/debian-amd64.docker       |   2 -
 tests/docker/dockerfiles/debian-arm64-cross.docker |   4 +-
 tests/docker/dockerfiles/debian-armel-cross.docker |   4 +-
 tests/docker/dockerfiles/debian-armhf-cross.docker |   4 +-
 .../dockerfiles/debian-mips64el-cross.docker       |   4 +-
 .../docker/dockerfiles/debian-mipsel-cross.docker  |   4 +-
 .../docker/dockerfiles/debian-ppc64el-cross.docker |   4 +-
 tests/docker/dockerfiles/debian-s390x-cross.docker |   4 +-
 tests/docker/dockerfiles/debian-toolchain.docker   |   1 -
 .../docker/dockerfiles/debian-tricore-cross.docker |   2 +-
 tests/docker/dockerfiles/fedora-win32-cross.docker |   6 +-
 tests/docker/dockerfiles/fedora-win64-cross.docker |   6 +-
 tests/docker/dockerfiles/fedora.docker             |   6 +-
 tests/docker/dockerfiles/opensuse-leap.docker      |   2 -
 tests/docker/dockerfiles/ubuntu2004.docker         |   2 -
 tests/lcitool/libvirt-ci                           |   2 +-
 tests/lcitool/projects/qemu.yml                    |   2 -
 tests/lcitool/refresh                              |   6 +-
 tests/qapi-schema/meson.build                      |   7 +-
 tests/tcg/Makefile.target                          |   4 +-
 tests/tcg/aarch64/Makefile.softmmu-target          |   7 +
 tests/tcg/aarch64/system/boot.S                    |   3 +-
 tests/tcg/multiarch/Makefile.target                |   9 +
 tests/vm/centos.aarch64                            |   2 +-
 74 files changed, 758 insertions(+), 272 deletions(-)
 create mode 100644 docs/about/emulation.rst
 create mode 100644 docs/system/introduction.rst
 delete mode 100644 docs/system/quickstart.rst
 create mode 100644 scripts/oss-fuzz/lsan_suppressions.txt
 delete mode 100644 scripts/shaderinclude.pl
 create mode 100644 scripts/shaderinclude.py

Comments

Peter Maydell Feb. 1, 2023, 4:15 p.m. UTC | #1
On Thu, 26 Jan 2023 at 11:22, Alex Bennée <alex.bennee@linaro.org> wrote:
>
> The following changes since commit 13356edb87506c148b163b8c7eb0695647d00c2a:
>
>   Merge tag 'block-pull-request' of https://gitlab.com/stefanha/qemu into staging (2023-01-24 09:45:33 +0000)
>
> are available in the Git repository at:
>
>   https://gitlab.com/stsquad/qemu.git tags/pull-jan-omnibus-260123-1
>
> for you to fetch changes up to d200653282f50df2a994a0609f88baa6a5889f80:
>
>   plugins: Iterate on cb_lists in qemu_plugin_user_exit (2023-01-26 11:12:01 +0000)
>
> ----------------------------------------------------------------
> Testing, docs, semihosting and plugin updates
>
>   - update playbooks for custom runners
>   - add section timing support to gitlab
>   - upgrade fedora images to 37
>   - purge perl from the build system and deps
>   - disable unstable tests in CI
>   - improve intro, emulation and semihosting docs
>   - semihosting bug fix and O_BINARY default
>   - add memory-sve test
>   - fix some races in qht
>   - improve plugin handling of memory helpers
>   - optimise plugin hooks
>   - fix some plugin deadlocks
>

Hi; this fails the docs build on macos 12 (probably a difference
in behaviour across Sphinx versions):
https://gitlab.com/qemu-project/qemu/-/jobs/3697858012

FAILED: docs/docs.stamp
/usr/bin/env CONFDIR=etc/qemu /opt/homebrew/bin/sphinx-build -q -W
-Dversion=7.2.50 -Drelease= -Ddepfile=docs/docs.d
-Ddepfile_stamp=docs/docs.stamp -b html -d
/private/var/folders/76/zy5ktkns50v6gt5g8r0sf6sc0000gn/T/cirrus-ci-build/build/docs/manual.p
/private/var/folders/76/zy5ktkns50v6gt5g8r0sf6sc0000gn/T/cirrus-ci-build/docs
/private/var/folders/76/zy5ktkns50v6gt5g8r0sf6sc0000gn/T/cirrus-ci-build/build/docs/manual
Warning, treated as error:
/private/var/folders/76/zy5ktkns50v6gt5g8r0sf6sc0000gn/T/cirrus-ci-build/qemu-options.hx:161:duplicate
label machine options, other instance in
/private/var/folders/76/zy5ktkns50v6gt5g8r0sf6sc0000gn/T/cirrus-ci-build/docs/system/invocation.rst

I think this is "you can't put labels in qemu-options.hx,
because it gets included in two .rst files (invocation.rst
and qemu-manpage.rst), and Sphinx complains about the
duplicate labels, even though one of the two files is
only used in the HTML and one is only used in the manpages".

thanks
-- PMM
Alex Bennée Feb. 1, 2023, 6:06 p.m. UTC | #2
Peter Maydell <peter.maydell@linaro.org> writes:

> On Thu, 26 Jan 2023 at 11:22, Alex Bennée <alex.bennee@linaro.org> wrote:
>>
>> The following changes since commit 13356edb87506c148b163b8c7eb0695647d00c2a:
>>
>>   Merge tag 'block-pull-request' of https://gitlab.com/stefanha/qemu
>> into staging (2023-01-24 09:45:33 +0000)
>>
>> are available in the Git repository at:
>>
>>   https://gitlab.com/stsquad/qemu.git tags/pull-jan-omnibus-260123-1
>>
>> for you to fetch changes up to d200653282f50df2a994a0609f88baa6a5889f80:
>>
>>   plugins: Iterate on cb_lists in qemu_plugin_user_exit (2023-01-26 11:12:01 +0000)
>>
>> ----------------------------------------------------------------
>> Testing, docs, semihosting and plugin updates
>>
>>   - update playbooks for custom runners
>>   - add section timing support to gitlab
>>   - upgrade fedora images to 37
>>   - purge perl from the build system and deps
>>   - disable unstable tests in CI
>>   - improve intro, emulation and semihosting docs
>>   - semihosting bug fix and O_BINARY default
>>   - add memory-sve test
>>   - fix some races in qht
>>   - improve plugin handling of memory helpers
>>   - optimise plugin hooks
>>   - fix some plugin deadlocks
>>
>
> Hi; this fails the docs build on macos 12 (probably a difference
> in behaviour across Sphinx versions):
> https://gitlab.com/qemu-project/qemu/-/jobs/3697858012
>
> FAILED: docs/docs.stamp
> /usr/bin/env CONFDIR=etc/qemu /opt/homebrew/bin/sphinx-build -q -W
> -Dversion=7.2.50 -Drelease= -Ddepfile=docs/docs.d
> -Ddepfile_stamp=docs/docs.stamp -b html -d
> /private/var/folders/76/zy5ktkns50v6gt5g8r0sf6sc0000gn/T/cirrus-ci-build/build/docs/manual.p
> /private/var/folders/76/zy5ktkns50v6gt5g8r0sf6sc0000gn/T/cirrus-ci-build/docs
> /private/var/folders/76/zy5ktkns50v6gt5g8r0sf6sc0000gn/T/cirrus-ci-build/build/docs/manual
> Warning, treated as error:
> /private/var/folders/76/zy5ktkns50v6gt5g8r0sf6sc0000gn/T/cirrus-ci-build/qemu-options.hx:161:duplicate
> label machine options, other instance in
> /private/var/folders/76/zy5ktkns50v6gt5g8r0sf6sc0000gn/T/cirrus-ci-build/docs/system/invocation.rst
>
> I think this is "you can't put labels in qemu-options.hx,
> because it gets included in two .rst files (invocation.rst
> and qemu-manpage.rst), and Sphinx complains about the
> duplicate labels, even though one of the two files is
> only used in the HTML and one is only used in the manpages".

Oh boo - anyway to work around that because they are helpful links?
Peter Maydell Feb. 2, 2023, 10:50 a.m. UTC | #3
On Wed, 1 Feb 2023 at 18:07, Alex Bennée <alex.bennee@linaro.org> wrote:
> Peter Maydell <peter.maydell@linaro.org> writes:
> > I think this is "you can't put labels in qemu-options.hx,
> > because it gets included in two .rst files (invocation.rst
> > and qemu-manpage.rst), and Sphinx complains about the
> > duplicate labels, even though one of the two files is
> > only used in the HTML and one is only used in the manpages".
>
> Oh boo - anyway to work around that because they are helpful links?

Nothing easy. The problem is that Sphinx looks at every .rst
file in the source directory, regardless of whether it's
reachable from the document you specify as the root of the
manual or not. So both lots of .rst files get processed
for both the HTML manual set and the manpages, even though
they don't need to be[*]. This is a long-standing design
deficiency in Sphinx. The only thing I could think of was
splitting the manpages and html docs entirely into separate
subdirectories, and having meson symlink the files which are
actually shared between them. But that seems like quite a lot
of extra machinery.

[*] This shows up for instance in the HTML docs getting a
not-linked-to-from-anywhere HTML version of the qemu(1) manpage:
https://www.qemu.org/docs/master/system/qemu-manpage.html

-- PMM
Alex Bennée Feb. 2, 2023, 5:12 p.m. UTC | #4
Peter Maydell <peter.maydell@linaro.org> writes:

> On Wed, 1 Feb 2023 at 18:07, Alex Bennée <alex.bennee@linaro.org> wrote:
>> Peter Maydell <peter.maydell@linaro.org> writes:
>> > I think this is "you can't put labels in qemu-options.hx,
>> > because it gets included in two .rst files (invocation.rst
>> > and qemu-manpage.rst), and Sphinx complains about the
>> > duplicate labels, even though one of the two files is
>> > only used in the HTML and one is only used in the manpages".
>>
>> Oh boo - anyway to work around that because they are helpful links?
>
> Nothing easy. The problem is that Sphinx looks at every .rst
> file in the source directory, regardless of whether it's
> reachable from the document you specify as the root of the
> manual or not. So both lots of .rst files get processed
> for both the HTML manual set and the manpages, even though
> they don't need to be[*]. This is a long-standing design
> deficiency in Sphinx. The only thing I could think of was
> splitting the manpages and html docs entirely into separate
> subdirectories, and having meson symlink the files which are
> actually shared between them. But that seems like quite a lot
> of extra machinery.

OK I've scrubbed the back links and sent v2 (which also reduces the
buildtime of cross-win64-system which kept tripping).

>
> [*] This shows up for instance in the HTML docs getting a
> not-linked-to-from-anywhere HTML version of the qemu(1) manpage:
> https://www.qemu.org/docs/master/system/qemu-manpage.html
>
> -- PMM