From patchwork Mon Oct 26 19:42:41 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 301950 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id ECF2BC4363A for ; Mon, 26 Oct 2020 19:58:00 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 4410C20795 for ; Mon, 26 Oct 2020 19:57:58 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="M8qLNjoW" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4410C20795 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:34690 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kX8cv-0005hh-6J for qemu-devel@archiver.kernel.org; Mon, 26 Oct 2020 15:57:57 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:33984) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kX8P4-00075P-P1 for qemu-devel@nongnu.org; Mon, 26 Oct 2020 15:43:39 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:53327) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.90_1) (envelope-from ) id 1kX8P0-0003BN-NN for qemu-devel@nongnu.org; Mon, 26 Oct 2020 15:43:38 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1603741412; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=1xfVczBZHXIOJicfDKEXSO22nRTr/xoCwuVo6EOzjJw=; b=M8qLNjoWD+ePzoFS9yIvRs3ndJmGl7agrfeKtOBl7wJENLm+lGWHKn1jdUVXv7jXRzjp1G Y66T3dllhz2R/KGy/W2j0FMc6MKGgQzDXt2aOsNk1eHI+La0sz23524gw45UJ1ONZO6lhu 5ScTcRQCHZ2c8FiV0TPD3cxIeFGnC3E= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-419-ccJTFC1kMUKFuW9z6xCplA-1; Mon, 26 Oct 2020 15:43:15 -0400 X-MC-Unique: ccJTFC1kMUKFuW9z6xCplA-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id D0C0C1087D96 for ; Mon, 26 Oct 2020 19:43:08 +0000 (UTC) Received: from scv.redhat.com (ovpn-118-238.rdu2.redhat.com [10.10.118.238]) by smtp.corp.redhat.com (Postfix) with ESMTP id ACC76925D6; Mon, 26 Oct 2020 19:43:07 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org, Markus Armbruster Subject: [PATCH v2 01/11] [DO-NOT-MERGE] docs: replace single backtick (`) with double-backtick (``) Date: Mon, 26 Oct 2020 15:42:41 -0400 Message-Id: <20201026194251.11075-2-jsnow@redhat.com> In-Reply-To: <20201026194251.11075-1-jsnow@redhat.com> References: <20201026194251.11075-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=jsnow@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=216.205.24.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/10/26 02:39:09 X-ACL-Warn: Detected OS = Linux 2.2.x-3.x [generic] [fuzzy] X-Spam_score_int: 0 X-Spam_score: -0.1 X-Spam_bar: / X-Spam_report: (-0.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, PDS_OTHER_BAD_TLD=1.999, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: John Snow , Eduardo Habkost , Cleber Rosa Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" The single backtick in ReST is the "default role". Currently, Sphinx's default role is called "content". Sphinx suggests you can use the "Any" role instead to turn any single-backtick enclosed item into a cross-reference. Before we do that, though, we'll need to turn all existing usages of the "content" role to inline verbatim markup by using double backticks instead. Signed-off-by: John Snow --- docs/devel/build-system.rst | 120 ++++++++++++------------- docs/devel/migration.rst | 59 ++++++------ docs/devel/tcg-plugins.rst | 14 +-- docs/devel/testing.rst | 2 +- docs/interop/live-block-operations.rst | 4 +- docs/system/arm/cpu-features.rst | 110 +++++++++++------------ docs/system/arm/nuvoton.rst | 2 +- docs/system/s390x/protvirt.rst | 10 +-- qapi/block-core.json | 4 +- 9 files changed, 164 insertions(+), 161 deletions(-) diff --git a/docs/devel/build-system.rst b/docs/devel/build-system.rst index 6fcf8854b70..eb7b7bbd3a3 100644 --- a/docs/devel/build-system.rst +++ b/docs/devel/build-system.rst @@ -53,14 +53,14 @@ following tasks: - Add a Meson build option to meson_options.txt. - Add support to the command line arg parser to handle any new - `--enable-XXX`/`--disable-XXX` flags required by the feature. + ``--enable-XXX``/``--disable-XXX`` flags required by the feature. - Add information to the help output message to report on the new feature flag. - Add code to perform the actual feature check. - - Add code to include the feature status in `config-host.h` + - Add code to include the feature status in ``config-host.h`` - Add code to print out the feature status in the configure summary upon completion. @@ -116,51 +116,51 @@ Helper functions The configure script provides a variety of helper functions to assist developers in checking for system features: -`do_cc $ARGS...` +``do_cc $ARGS...`` Attempt to run the system C compiler passing it $ARGS... -`do_cxx $ARGS...` +``do_cxx $ARGS...`` Attempt to run the system C++ compiler passing it $ARGS... -`compile_object $CFLAGS` +``compile_object $CFLAGS`` Attempt to compile a test program with the system C compiler using $CFLAGS. The test program must have been previously written to a file - called $TMPC. The replacement in Meson is the compiler object `cc`, - which has methods such as `cc.compiles()`, - `cc.check_header()`, `cc.has_function()`. + called $TMPC. The replacement in Meson is the compiler object ``cc``, + which has methods such as ``cc.compiles()``, + ``cc.check_header()``, ``cc.has_function()``. -`compile_prog $CFLAGS $LDFLAGS` +``compile_prog $CFLAGS $LDFLAGS`` Attempt to compile a test program with the system C compiler using $CFLAGS and link it with the system linker using $LDFLAGS. The test program must have been previously written to a file called $TMPC. - The replacement in Meson is `cc.find_library()` and `cc.links()`. + The replacement in Meson is ``cc.find_library()`` and ``cc.links()``. -`has $COMMAND` +``has $COMMAND`` Determine if $COMMAND exists in the current environment, either as a shell builtin, or executable binary, returning 0 on success. The - replacement in Meson is `find_program()`. + replacement in Meson is ``find_program()``. -`check_define $NAME` +``check_define $NAME`` Determine if the macro $NAME is defined by the system C compiler -`check_include $NAME` +``check_include $NAME`` Determine if the include $NAME file is available to the system C - compiler. The replacement in Meson is `cc.has_header()`. + compiler. The replacement in Meson is ``cc.has_header()``. -`write_c_skeleton` +``write_c_skeleton`` Write a minimal C program main() function to the temporary file indicated by $TMPC -`feature_not_found $NAME $REMEDY` +``feature_not_found $NAME $REMEDY`` Print a message to stderr that the feature $NAME was not available on the system, suggesting the user try $REMEDY to address the problem. -`error_exit $MESSAGE $MORE...` +``error_exit $MESSAGE $MORE...`` Print $MESSAGE to stderr, followed by $MORE... and then exit from the configure script with non-zero status -`query_pkg_config $ARGS...` +``query_pkg_config $ARGS...`` Run pkg-config passing it $ARGS. If QEMU is doing a static build, then --static will be automatically added to $ARGS @@ -193,14 +193,14 @@ compilation as possible. The Meson "sourceset" functionality is used to list the files and their dependency on various configuration symbols. -All executables are built by default, except for some `contrib/` +All executables are built by default, except for some ``contrib/`` binaries that are known to fail to build on some platforms (for example 32-bit or big-endian platforms). Tests are also built by default, though that might change in the future. Various subsystems that are common to both tools and emulators have -their own sourceset, for example `block_ss` for the block device subsystem, -`chardev_ss` for the character device subsystem, etc. These sourcesets +their own sourceset, for example ``block_ss`` for the block device subsystem, +``chardev_ss`` for the character device subsystem, etc. These sourcesets are then turned into static libraries as follows:: libchardev = static_library('chardev', chardev_ss.sources(), @@ -209,8 +209,8 @@ are then turned into static libraries as follows:: chardev = declare_dependency(link_whole: libchardev) -As of Meson 0.55.1, the special `.fa` suffix should be used for everything -that is used with `link_whole`, to ensure that the link flags are placed +As of Meson 0.55.1, the special ``.fa`` suffix should be used for everything +that is used with ``link_whole``, to ensure that the link flags are placed correctly in the command line. Files linked into emulator targets there can be split into two distinct groups @@ -221,25 +221,25 @@ In the target-independent set lives various general purpose helper code, such as error handling infrastructure, standard data structures, platform portability wrapper functions, etc. This code can be compiled once only and the .o files linked into all output binaries. -Target-independent code lives in the `common_ss`, `softmmu_ss` and -`user_ss` sourcesets. `common_ss` is linked into all emulators, `softmmu_ss` -only in system emulators, `user_ss` only in user-mode emulators. +Target-independent code lives in the ``common_ss``, ``softmmu_ss`` and +``user_ss`` sourcesets. ``common_ss`` is linked into all emulators, ``softmmu_ss`` +only in system emulators, ``user_ss`` only in user-mode emulators. In the target-dependent set lives CPU emulation, device emulation and much glue code. This sometimes also has to be compiled multiple times, once for each target being built. Target-dependent files are included -in the `specific_ss` sourceset. +in the ``specific_ss`` sourceset. -All binaries link with a static library `libqemuutil.a`, which is then -linked to all the binaries. `libqemuutil.a` is built from several +All binaries link with a static library ``libqemuutil.a``, which is then +linked to all the binaries. ``libqemuutil.a`` is built from several sourcesets; most of them however host generated code, and the only two -of general interest are `util_ss` and `stub_ss`. +of general interest are ``util_ss`` and ``stub_ss``. The separation between these two is purely for documentation purposes. -`util_ss` contains generic utility files. Even though this code is only +``util_ss`` contains generic utility files. Even though this code is only linked in some binaries, sometimes it requires hooks only in some of these and depend on other functions that are not fully implemented by -all QEMU binaries. `stub_ss` links dummy stubs that will only be linked +all QEMU binaries. ``stub_ss`` links dummy stubs that will only be linked into the binary if the real implementation is not present. In a way, the stubs can be thought of as a portable implementation of the weak symbols concept. @@ -247,7 +247,7 @@ symbols concept. The following files concur in the definition of which files are linked into each emulator: -`default-configs/*.mak` +``default-configs/*.mak`` The files under default-configs/ control what emulated hardware is built into each QEMU system and userspace emulator targets. They merely contain a list of config variable definitions like the machines that should be @@ -257,8 +257,8 @@ into each emulator: CONFIG_XLNX_ZYNQMP_ARM=y CONFIG_XLNX_VERSAL=y -`*/Kconfig` - These files are processed together with `default-configs/*.mak` and +``*/Kconfig`` + These files are processed together with ``default-configs/*.mak`` and describe the dependencies between various features, subsystems and device models. They are described in kconfig.rst. @@ -270,19 +270,19 @@ Support scripts --------------- Meson has a special convention for invoking Python scripts: if their -first line is `#! /usr/bin/env python3` and the file is *not* executable, +first line is ``#! /usr/bin/env python3`` and the file is *not* executable, find_program() arranges to invoke the script under the same Python interpreter that was used to invoke Meson. This is the most common and preferred way to invoke support scripts from Meson build files, because it automatically uses the value of configure's --python= option. -In case the script is not written in Python, use a `#! /usr/bin/env ...` +In case the script is not written in Python, use a ``#! /usr/bin/env ...`` line and make the script executable. Scripts written in Python, where it is desirable to make the script executable (for example for test scripts that developers may want to invoke from the command line, such as tests/qapi-schema/test-qapi.py), -should be invoked through the `python` variable in meson.build. For +should be invoked through the ``python`` variable in meson.build. For example:: test('QAPI schema regression tests', python, @@ -306,10 +306,10 @@ rules and wraps them so that e.g. submodules are built before QEMU. The resulting build system is largely non-recursive in nature, in contrast to common practices seen with automake. -Tests are also ran by the Makefile with the traditional `make check` -phony target, while benchmarks are run with `make bench`. Meson test -suites such as `unit` can be ran with `make check-unit` too. It is also -possible to run tests defined in meson.build with `meson test`. +Tests are also ran by the Makefile with the traditional ``make check`` +phony target, while benchmarks are run with ``make bench``. Meson test +suites such as ``unit`` can be ran with ``make check-unit`` too. It is also +possible to run tests defined in meson.build with ``meson test``. Important files for the build system ==================================== @@ -321,28 +321,28 @@ The following key files are statically defined in the source tree, with the rules needed to build QEMU. Their behaviour is influenced by a number of dynamically created files listed later. -`Makefile` +``Makefile`` The main entry point used when invoking make to build all the components of QEMU. The default 'all' target will naturally result in the build of every component. Makefile takes care of recursively building submodules directly via a non-recursive set of rules. -`*/meson.build` +``*/meson.build`` The meson.build file in the root directory is the main entry point for the Meson build system, and it coordinates the configuration and build of all executables. Build rules for various subdirectories are included in other meson.build files spread throughout the QEMU source tree. -`tests/Makefile.include` +``tests/Makefile.include`` Rules for external test harnesses. These include the TCG tests, - `qemu-iotests` and the Avocado-based acceptance tests. + ``qemu-iotests`` and the Avocado-based acceptance tests. -`tests/docker/Makefile.include` +``tests/docker/Makefile.include`` Rules for Docker tests. Like tests/Makefile, this file is included directly by the top level Makefile, anything defined in this file will influence the entire build system. -`tests/vm/Makefile.include` +``tests/vm/Makefile.include`` Rules for VM-based tests. Like tests/Makefile, this file is included directly by the top level Makefile, anything defined in this file will influence the entire build system. @@ -358,11 +358,11 @@ Makefile. Built by configure: -`config-host.mak` +``config-host.mak`` When configure has determined the characteristics of the build host it will write a long list of variables to config-host.mak file. This provides the various install directories, compiler / linker flags and a - variety of `CONFIG_*` variables related to optionally enabled features. + variety of ``CONFIG_*`` variables related to optionally enabled features. This is imported by the top level Makefile and meson.build in order to tailor the build output. @@ -374,7 +374,7 @@ Built by configure: build outputs. Variables which are potentially different for each emulator target are defined by the next file... -`$TARGET-NAME/config-target.mak` +``$TARGET-NAME/config-target.mak`` TARGET-NAME is the name of a system or userspace emulator, for example, x86_64-softmmu denotes the system emulator for the x86_64 architecture. This file contains the variables which need to vary on a per-target @@ -385,29 +385,29 @@ Built by configure: Built by Meson: -`${TARGET-NAME}-config-devices.mak` +``${TARGET-NAME}-config-devices.mak`` TARGET-NAME is again the name of a system or userspace emulator. The config-devices.mak file is automatically generated by make using the scripts/make_device_config.sh program, feeding it the default-configs/$TARGET-NAME file as input. -`config-host.h`, `$TARGET-NAME/config-target.h`, `$TARGET-NAME/config-devices.h` +``config-host.h``, ``$TARGET-NAME/config-target.h``, ``$TARGET-NAME/config-devices.h`` These files are used by source code to determine what features are enabled. They are generated from the contents of the corresponding - `*.h` files using the scripts/create_config program. This extracts + ``*.h`` files using the scripts/create_config program. This extracts relevant variables and formats them as C preprocessor macros. -`build.ninja` +``build.ninja`` The build rules. Built by Makefile: -`Makefile.ninja` +``Makefile.ninja`` A Makefile include that bridges to ninja for the actual build. The Makefile is mostly a list of targets that Meson included in build.ninja. -`Makefile.mtest` +``Makefile.mtest`` The Makefile definitions that let "make check" run tests defined in meson.build. The rules are produced from Meson's JSON description of tests (obtained with "meson introspect --tests") through the script @@ -417,9 +417,9 @@ Built by Makefile: Useful make targets ------------------- -`help` +``help`` Print a help message for the most common build targets. -`print-VAR` +``print-VAR`` Print the value of the variable VAR. Useful for debugging the build system. diff --git a/docs/devel/migration.rst b/docs/devel/migration.rst index 49112bb27aa..f50e1250359 100644 --- a/docs/devel/migration.rst +++ b/docs/devel/migration.rst @@ -53,7 +53,7 @@ savevm/loadvm functionality. Debugging ========= -The migration stream can be analyzed thanks to `scripts/analyze_migration.py`. +The migration stream can be analyzed thanks to ``scripts/analyze_migration.py``. Example usage: @@ -74,8 +74,8 @@ Common infrastructure ===================== The files, sockets or fd's that carry the migration stream are abstracted by -the ``QEMUFile`` type (see `migration/qemu-file.h`). In most cases this -is connected to a subtype of ``QIOChannel`` (see `io/`). +the ``QEMUFile`` type (see ``migration/qemu-file.h``). In most cases this +is connected to a subtype of ``QIOChannel`` (see ``io/``). Saving the state of one device @@ -165,14 +165,14 @@ An example (from hw/input/pckbd.c) }; We are declaring the state with name "pckbd". -The `version_id` is 3, and the fields are 4 uint8_t in a KBDState structure. +The ``version_id`` is 3, and the fields are 4 uint8_t in a KBDState structure. We registered this with: .. code:: c vmstate_register(NULL, 0, &vmstate_kbd, s); -For devices that are `qdev` based, we can register the device in the class +For devices that are ``qdev`` based, we can register the device in the class init function: .. code:: c @@ -209,9 +209,9 @@ another to load the state back. SaveVMHandlers *ops, void *opaque); -Two functions in the ``ops`` structure are the `save_state` -and `load_state` functions. Notice that `load_state` receives a version_id -parameter to know what state format is receiving. `save_state` doesn't +Two functions in the ``ops`` structure are the ``save_state`` +and ``load_state`` functions. Notice that ``load_state`` receives a version_id +parameter to know what state format is receiving. ``save_state`` doesn't have a version_id parameter because it always uses the latest version. Note that because the VMState macros still save the data in a raw @@ -384,26 +384,28 @@ migration of a device, and using them breaks backward-migration compatibility; in general most changes can be made by adding Subsections (see above) or _TEST macros (see above) which won't break compatibility. -Each version is associated with a series of fields saved. The `save_state` always saves -the state as the newer version. But `load_state` sometimes is able to -load state from an older version. +Each version is associated with a series of fields saved. The +``save_state`` always saves the state as the newer version. But +``load_state`` sometimes is able to load state from an older version. You can see that there are several version fields: -- `version_id`: the maximum version_id supported by VMState for that device. -- `minimum_version_id`: the minimum version_id that VMState is able to understand - for that device. -- `minimum_version_id_old`: For devices that were not able to port to vmstate, we can - assign a function that knows how to read this old state. This field is - ignored if there is no `load_state_old` handler. +- ``version_id``: the maximum version_id supported by VMState for that + device. +- ``minimum_version_id``: the minimum version_id that VMState is able to + understand for that device. +- ``minimum_version_id_old``: For devices that were not able to port to + vmstate, we can assign a function that knows how to read this old + state. This field is ignored if there is no ``load_state_old`` + handler. VMState is able to read versions from minimum_version_id to version_id. And the function ``load_state_old()`` (if present) is able to load state from minimum_version_id_old to minimum_version_id. This function is deprecated and will be removed when no more users are left. -There are *_V* forms of many ``VMSTATE_`` macros to load fields for version dependent fields, -e.g. +There are *_V* forms of many ``VMSTATE_`` macros to load fields for +version dependent fields, e.g. .. code:: c @@ -453,7 +455,7 @@ data and then transferred to the main structure. If you use memory API functions that update memory layout outside initialization (i.e., in response to a guest action), this is a strong -indication that you need to call these functions in a `post_load` callback. +indication that you need to call these functions in a ``post_load`` callback. Examples of such memory API functions are: - memory_region_add_subregion() @@ -818,17 +820,18 @@ Postcopy now works with hugetlbfs backed memory: Postcopy with shared memory --------------------------- -Postcopy migration with shared memory needs explicit support from the other -processes that share memory and from QEMU. There are restrictions on the type of -memory that userfault can support shared. +Postcopy migration with shared memory needs explicit support from the +other processes that share memory and from QEMU. There are restrictions +on the type of memory that userfault can support shared. -The Linux kernel userfault support works on `/dev/shm` memory and on `hugetlbfs` -(although the kernel doesn't provide an equivalent to `madvise(MADV_DONTNEED)` -for hugetlbfs which may be a problem in some configurations). +The Linux kernel userfault support works on ``/dev/shm`` memory and on +``hugetlbfs`` (although the kernel doesn't provide an equivalent to +``madvise(MADV_DONTNEED)`` for hugetlbfs which may be a problem in some +configurations). The vhost-user code in QEMU supports clients that have Postcopy support, -and the `vhost-user-bridge` (in `tests/`) and the DPDK package have changes -to support postcopy. +and the ``vhost-user-bridge`` (in ``tests/``) and the DPDK package have +changes to support postcopy. The client needs to open a userfaultfd and register the areas of memory that it maps with userfault. The client must then pass the diff --git a/docs/devel/tcg-plugins.rst b/docs/devel/tcg-plugins.rst index 0568dfa6a49..5057b6e1b2b 100644 --- a/docs/devel/tcg-plugins.rst +++ b/docs/devel/tcg-plugins.rst @@ -34,11 +34,11 @@ version they were built against. This can be done simply by:: QEMU_PLUGIN_EXPORT int qemu_plugin_version = QEMU_PLUGIN_VERSION; The core code will refuse to load a plugin that doesn't export a -`qemu_plugin_version` symbol or if plugin version is outside of QEMU's +``qemu_plugin_version`` symbol or if plugin version is outside of QEMU's supported range of API versions. -Additionally the `qemu_info_t` structure which is passed to the -`qemu_plugin_install` method of a plugin will detail the minimum and +Additionally the ``qemu_info_t`` structure which is passed to the +``qemu_plugin_install`` method of a plugin will detail the minimum and current API versions supported by QEMU. The API version will be incremented if new APIs are added. The minimum API version will be incremented if existing APIs are changed or removed. @@ -140,12 +140,12 @@ Example Plugins There are a number of plugins included with QEMU and you are encouraged to contribute your own plugins plugins upstream. There is a -`contrib/plugins` directory where they can go. +``contrib/plugins`` directory where they can go. - tests/plugins These are some basic plugins that are used to test and exercise the -API during the `make check-tcg` target. +API during the ``make check-tcg`` target. - contrib/plugins/hotblocks.c @@ -157,7 +157,7 @@ with linux-user execution as system emulation tends to generate re-translations as blocks from different programs get swapped in and out of system memory. -If your program is single-threaded you can use the `inline` option for +If your program is single-threaded you can use the ``inline`` option for slightly faster (but not thread safe) counters. Example:: @@ -245,7 +245,7 @@ which will lead to a sorted list after the class breakdown:: ... To find the argument shorthand for the class you need to examine the -source code of the plugin at the moment, specifically the `*opt` +source code of the plugin at the moment, specifically the ``*opt`` argument in the InsnClassExecCount tables. - contrib/plugins/lockstep.c diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst index 0c3e79d31cd..4ab84926cb2 100644 --- a/docs/devel/testing.rst +++ b/docs/devel/testing.rst @@ -707,7 +707,7 @@ The base test class has also support for tests with more than one QEMUMachine. The way to get machines is through the ``self.get_vm()`` method which will return a QEMUMachine instance. The ``self.get_vm()`` method accepts arguments that will be passed to the QEMUMachine creation -and also an optional `name` attribute so you can identify a specific +and also an optional ``name`` attribute so you can identify a specific machine and get it more than once through the tests methods. A simple and hypothetical example follows: diff --git a/docs/interop/live-block-operations.rst b/docs/interop/live-block-operations.rst index e13f5a21f8d..5a3f0458275 100644 --- a/docs/interop/live-block-operations.rst +++ b/docs/interop/live-block-operations.rst @@ -638,7 +638,7 @@ at this point: (QEMU) block-job-complete device=job0 In either of the above cases, if you once again run the -`query-block-jobs` command, there should not be any active block +``query-block-jobs`` command, there should not be any active block operation. Comparing 'commit' and 'mirror': In both then cases, the overlay images @@ -777,7 +777,7 @@ the content of image [D]. } (6) [On *destination* QEMU] Finally, resume the guest vCPUs by issuing the - QMP command `cont`:: + QMP command ``cont``:: (QEMU) cont { diff --git a/docs/system/arm/cpu-features.rst b/docs/system/arm/cpu-features.rst index 35196a6b759..3622c904798 100644 --- a/docs/system/arm/cpu-features.rst +++ b/docs/system/arm/cpu-features.rst @@ -10,22 +10,22 @@ is the Performance Monitoring Unit (PMU). CPU types such as the Cortex-A15 and the Cortex-A57, which respectively implement Arm architecture reference manuals ARMv7-A and ARMv8-A, may both optionally implement PMUs. For example, if a user wants to use a Cortex-A15 without -a PMU, then the `-cpu` parameter should contain `pmu=off` on the QEMU -command line, i.e. `-cpu cortex-a15,pmu=off`. +a PMU, then the ``-cpu`` parameter should contain ``pmu=off`` on the QEMU +command line, i.e. ``-cpu cortex-a15,pmu=off``. As not all CPU types support all optional CPU features, then whether or not a CPU property exists depends on the CPU type. For example, CPUs that implement the ARMv8-A architecture reference manual may optionally support the AArch32 CPU feature, which may be enabled by disabling the -`aarch64` CPU property. A CPU type such as the Cortex-A15, which does -not implement ARMv8-A, will not have the `aarch64` CPU property. +``aarch64`` CPU property. A CPU type such as the Cortex-A15, which does +not implement ARMv8-A, will not have the ``aarch64`` CPU property. QEMU's support may be limited for some CPU features, only partially supporting the feature or only supporting the feature under certain -configurations. For example, the `aarch64` CPU feature, which, when +configurations. For example, the ``aarch64`` CPU feature, which, when disabled, enables the optional AArch32 CPU feature, is only supported when using the KVM accelerator and when running on a host CPU type that -supports the feature. While `aarch64` currently only works with KVM, +supports the feature. While ``aarch64`` currently only works with KVM, it could work with TCG. CPU features that are specific to KVM are prefixed with "kvm-" and are described in "KVM VCPU Features". @@ -33,12 +33,12 @@ CPU Feature Probing =================== Determining which CPU features are available and functional for a given -CPU type is possible with the `query-cpu-model-expansion` QMP command. -Below are some examples where `scripts/qmp/qmp-shell` (see the top comment +CPU type is possible with the ``query-cpu-model-expansion`` QMP command. +Below are some examples where ``scripts/qmp/qmp-shell`` (see the top comment block in the script for usage) is used to issue the QMP commands. -1. Determine which CPU features are available for the `max` CPU type - (Note, we started QEMU with qemu-system-aarch64, so `max` is +1. Determine which CPU features are available for the ``max`` CPU type + (Note, we started QEMU with qemu-system-aarch64, so ``max`` is implementing the ARMv8-A reference manual in this case):: (QEMU) query-cpu-model-expansion type=full model={"name":"max"} @@ -51,9 +51,9 @@ block in the script for usage) is used to issue the QMP commands. "sve896": true, "sve1280": true, "sve2048": true }}}} -We see that the `max` CPU type has the `pmu`, `aarch64`, `sve`, and many -`sve` CPU features. We also see that all the CPU features are -enabled, as they are all `true`. (The `sve` CPU features are all +We see that the ``max`` CPU type has the ``pmu``, ``aarch64``, ``sve``, and many +``sve`` CPU features. We also see that all the CPU features are +enabled, as they are all ``true``. (The ``sve`` CPU features are all optional SVE vector lengths (see "SVE CPU Properties"). While with TCG all SVE vector lengths can be supported, when KVM is in use it's more likely that only a few lengths will be supported, if SVE is supported at @@ -71,9 +71,9 @@ all.) "sve896": true, "sve1280": true, "sve2048": true }}}} -We see it worked, as `pmu` is now `false`. +We see it worked, as ``pmu`` is now ``false``. -(3) Let's try to disable `aarch64`, which enables the AArch32 CPU feature:: +(3) Let's try to disable ``aarch64``, which enables the AArch32 CPU feature:: (QEMU) query-cpu-model-expansion type=full model={"name":"max","props":{"aarch64":false}} {"error": { @@ -84,7 +84,7 @@ We see it worked, as `pmu` is now `false`. It looks like this feature is limited to a configuration we do not currently have. -(4) Let's disable `sve` and see what happens to all the optional SVE +(4) Let's disable ``sve`` and see what happens to all the optional SVE vector lengths:: (QEMU) query-cpu-model-expansion type=full model={"name":"max","props":{"sve":false}} @@ -97,14 +97,14 @@ currently have. "sve896": false, "sve1280": false, "sve2048": false }}}} -As expected they are now all `false`. +As expected they are now all ``false``. (5) Let's try probing CPU features for the Cortex-A15 CPU type:: (QEMU) query-cpu-model-expansion type=full model={"name":"cortex-a15"} {"return": {"model": {"name": "cortex-a15", "props": {"pmu": true}}}} -Only the `pmu` CPU feature is available. +Only the ``pmu`` CPU feature is available. A note about CPU feature dependencies ------------------------------------- @@ -123,29 +123,29 @@ A note about CPU models and KVM ------------------------------- Named CPU models generally do not work with KVM. There are a few cases -that do work, e.g. using the named CPU model `cortex-a57` with KVM on a -seattle host, but mostly if KVM is enabled the `host` CPU type must be +that do work, e.g. using the named CPU model ``cortex-a57`` with KVM on a +seattle host, but mostly if KVM is enabled the ``host`` CPU type must be used. This means the guest is provided all the same CPU features as the -host CPU type has. And, for this reason, the `host` CPU type should +host CPU type has. And, for this reason, the ``host`` CPU type should enable all CPU features that the host has by default. Indeed it's even a bit strange to allow disabling CPU features that the host has when using -the `host` CPU type, but in the absence of CPU models it's the best we can +the ``host`` CPU type, but in the absence of CPU models it's the best we can do if we want to launch guests without all the host's CPU features enabled. -Enabling KVM also affects the `query-cpu-model-expansion` QMP command. The +Enabling KVM also affects the ``query-cpu-model-expansion`` QMP command. The affect is not only limited to specific features, as pointed out in example (3) of "CPU Feature Probing", but also to which CPU types may be expanded. -When KVM is enabled, only the `max`, `host`, and current CPU type may be +When KVM is enabled, only the ``max``, ``host``, and current CPU type may be expanded. This restriction is necessary as it's not possible to know all CPU types that may work with KVM, but it does impose a small risk of users experiencing unexpected errors. For example on a seattle, as mentioned -above, the `cortex-a57` CPU type is also valid when KVM is enabled. -Therefore a user could use the `host` CPU type for the current type, but -then attempt to query `cortex-a57`, however that query will fail with our +above, the ``cortex-a57`` CPU type is also valid when KVM is enabled. +Therefore a user could use the ``host`` CPU type for the current type, but +then attempt to query ``cortex-a57``, however that query will fail with our restrictions. This shouldn't be an issue though as management layers and -users have been preferring the `host` CPU type for use with KVM for quite +users have been preferring the ``host`` CPU type for use with KVM for quite some time. Additionally, if the KVM-enabled QEMU instance running on a -seattle host is using the `cortex-a57` CPU type, then querying `cortex-a57` +seattle host is using the ``cortex-a57`` CPU type, then querying ``cortex-a57`` will work. Using CPU Features @@ -158,12 +158,12 @@ QEMU command line with that CPU type:: $ qemu-system-aarch64 -M virt -cpu max,pmu=off,sve=on,sve128=on,sve256=on The example above disables the PMU and enables the first two SVE vector -lengths for the `max` CPU type. Note, the `sve=on` isn't actually -necessary, because, as we observed above with our probe of the `max` CPU -type, `sve` is already on by default. Also, based on our probe of +lengths for the ``max`` CPU type. Note, the ``sve=on`` isn't actually +necessary, because, as we observed above with our probe of the ``max`` CPU +type, ``sve`` is already on by default. Also, based on our probe of defaults, it would seem we need to disable many SVE vector lengths, rather than only enabling the two we want. This isn't the case, because, as -disabling many SVE vector lengths would be quite verbose, the `sve` CPU +disabling many SVE vector lengths would be quite verbose, the ``sve`` CPU properties have special semantics (see "SVE CPU Property Parsing Semantics"). @@ -214,49 +214,49 @@ the list of KVM VCPU features and their descriptions. SVE CPU Properties ================== -There are two types of SVE CPU properties: `sve` and `sve`. The first -is used to enable or disable the entire SVE feature, just as the `pmu` +There are two types of SVE CPU properties: ``sve`` and ``sve``. The first +is used to enable or disable the entire SVE feature, just as the ``pmu`` CPU property completely enables or disables the PMU. The second type -is used to enable or disable specific vector lengths, where `N` is the -number of bits of the length. The `sve` CPU properties have special +is used to enable or disable specific vector lengths, where ``N`` is the +number of bits of the length. The ``sve`` CPU properties have special dependencies and constraints, see "SVE CPU Property Dependencies and Constraints" below. Additionally, as we want all supported vector lengths to be enabled by default, then, in order to avoid overly verbose command -lines (command lines full of `sve=off`, for all `N` not wanted), we +lines (command lines full of ``sve=off``, for all ``N`` not wanted), we provide the parsing semantics listed in "SVE CPU Property Parsing Semantics". SVE CPU Property Dependencies and Constraints --------------------------------------------- - 1) At least one vector length must be enabled when `sve` is enabled. + 1) At least one vector length must be enabled when ``sve`` is enabled. - 2) If a vector length `N` is enabled, then, when KVM is enabled, all + 2) If a vector length ``N`` is enabled, then, when KVM is enabled, all smaller, host supported vector lengths must also be enabled. If KVM is not enabled, then only all the smaller, power-of-two vector lengths must be enabled. E.g. with KVM if the host supports all - vector lengths up to 512-bits (128, 256, 384, 512), then if `sve512` + vector lengths up to 512-bits (128, 256, 384, 512), then if ``sve512`` is enabled, the 128-bit vector length, 256-bit vector length, and 384-bit vector length must also be enabled. Without KVM, the 384-bit vector length would not be required. 3) If KVM is enabled then only vector lengths that the host CPU type support may be enabled. If SVE is not supported by the host, then - no `sve*` properties may be enabled. + no ``sve*`` properties may be enabled. SVE CPU Property Parsing Semantics ---------------------------------- - 1) If SVE is disabled (`sve=off`), then which SVE vector lengths + 1) If SVE is disabled (``sve=off``), then which SVE vector lengths are enabled or disabled is irrelevant to the guest, as the entire SVE feature is disabled and that disables all vector lengths for - the guest. However QEMU will still track any `sve` CPU - properties provided by the user. If later an `sve=on` is provided, - then the guest will get only the enabled lengths. If no `sve=on` + the guest. However QEMU will still track any ``sve`` CPU + properties provided by the user. If later an ``sve=on`` is provided, + then the guest will get only the enabled lengths. If no ``sve=on`` is provided and there are explicitly enabled vector lengths, then an error is generated. - 2) If SVE is enabled (`sve=on`), but no `sve` CPU properties are + 2) If SVE is enabled (``sve=on``), but no ``sve`` CPU properties are provided, then all supported vector lengths are enabled, which when KVM is not in use means including the non-power-of-two lengths, and, when KVM is in use, it means all vector lengths supported by the host @@ -272,7 +272,7 @@ SVE CPU Property Parsing Semantics constraint (2) of "SVE CPU Property Dependencies and Constraints"). 5) When KVM is enabled, if the host does not support SVE, then an error - is generated when attempting to enable any `sve*` properties (see + is generated when attempting to enable any ``sve*`` properties (see constraint (3) of "SVE CPU Property Dependencies and Constraints"). 6) When KVM is enabled, if the host does support SVE, then an error is @@ -280,8 +280,8 @@ SVE CPU Property Parsing Semantics by the host (see constraint (3) of "SVE CPU Property Dependencies and Constraints"). - 7) If one or more `sve` CPU properties are set `off`, but no `sve`, - CPU properties are set `on`, then the specified vector lengths are + 7) If one or more ``sve`` CPU properties are set ``off``, but no ``sve``, + CPU properties are set ``on``, then the specified vector lengths are disabled but the default for any unspecified lengths remains enabled. When KVM is not enabled, disabling a power-of-two vector length also disables all vector lengths larger than the power-of-two length. @@ -289,15 +289,15 @@ SVE CPU Property Parsing Semantics disables all larger vector lengths (see constraint (2) of "SVE CPU Property Dependencies and Constraints"). - 8) If one or more `sve` CPU properties are set to `on`, then they + 8) If one or more ``sve`` CPU properties are set to ``on``, then they are enabled and all unspecified lengths default to disabled, except for the required lengths per constraint (2) of "SVE CPU Property Dependencies and Constraints", which will even be auto-enabled if they were not explicitly enabled. - 9) If SVE was disabled (`sve=off`), allowing all vector lengths to be + 9) If SVE was disabled (``sve=off``), allowing all vector lengths to be explicitly disabled (i.e. avoiding the error specified in (3) of - "SVE CPU Property Parsing Semantics"), then if later an `sve=on` is + "SVE CPU Property Parsing Semantics"), then if later an ``sve=on`` is provided an error will be generated. To avoid this error, one must enable at least one vector length prior to enabling SVE. @@ -308,12 +308,12 @@ SVE CPU Property Examples $ qemu-system-aarch64 -M virt -cpu max,sve=off - 2) Implicitly enable all vector lengths for the `max` CPU type:: + 2) Implicitly enable all vector lengths for the ``max`` CPU type:: $ qemu-system-aarch64 -M virt -cpu max 3) When KVM is enabled, implicitly enable all host CPU supported vector - lengths with the `host` CPU type:: + lengths with the ``host`` CPU type:: $ qemu-system-aarch64 -M virt,accel=kvm -cpu host diff --git a/docs/system/arm/nuvoton.rst b/docs/system/arm/nuvoton.rst index e3e1a3a3a73..b0a35b54bd4 100644 --- a/docs/system/arm/nuvoton.rst +++ b/docs/system/arm/nuvoton.rst @@ -77,7 +77,7 @@ Boot options ------------ The Nuvoton machines can boot from an OpenBMC firmware image, or directly into -a kernel using the ``-kernel`` option. OpenBMC images for `quanta-gsj` and +a kernel using the ``-kernel`` option. OpenBMC images for ``quanta-gsj`` and possibly others can be downloaded from the OpenPOWER jenkins : https://openpower.xyz/ diff --git a/docs/system/s390x/protvirt.rst b/docs/system/s390x/protvirt.rst index 712974ad87b..d208c12a962 100644 --- a/docs/system/s390x/protvirt.rst +++ b/docs/system/s390x/protvirt.rst @@ -14,11 +14,11 @@ Prerequisites To run PVMs, a machine with the Protected Virtualization feature, as indicated by the Ultravisor Call facility (stfle bit 158), is required. The Ultravisor needs to be initialized at boot by setting -`prot_virt=1` on the host's kernel command line. +``prot_virt=1`` on the host's kernel command line. Running PVMs requires using the KVM hypervisor. -If those requirements are met, the capability `KVM_CAP_S390_PROTECTED` +If those requirements are met, the capability ``KVM_CAP_S390_PROTECTED`` will indicate that KVM can support PVMs on that LPAR. @@ -26,8 +26,8 @@ QEMU Settings ------------- To indicate to the VM that it can transition into protected mode, the -`Unpack facility` (stfle bit 161 represented by the feature -`unpack`/`S390_FEAT_UNPACK`) needs to be part of the cpu model of +``Unpack facility`` (stfle bit 161 represented by the feature +``unpack``/``S390_FEAT_UNPACK``) needs to be part of the cpu model of the VM. All I/O devices need to use the IOMMU. @@ -56,5 +56,5 @@ from the disk boot. This memory layout includes the encrypted components (kernel, initrd, cmdline), the stage3a loader and metadata. In case this boot method is used, the command line options -initrd and -cmdline are ineffective. The preparation of a PVM -image is done via the `genprotimg` tool from the s390-tools +image is done via the ``genprotimg`` tool from the s390-tools collection. diff --git a/qapi/block-core.json b/qapi/block-core.json index e00fc27b5ea..4ee81d226a9 100644 --- a/qapi/block-core.json +++ b/qapi/block-core.json @@ -498,11 +498,11 @@ # @status: current status of the dirty bitmap (since 2.4) # # @recording: true if the bitmap is recording new writes from the guest. -# Replaces `active` and `disabled` statuses. (since 4.0) +# Replaces ``active`` and ``disabled`` statuses. (since 4.0) # # @busy: true if the bitmap is in-use by some operation (NBD or jobs) # and cannot be modified via QMP or used by another operation. -# Replaces `locked` and `frozen` statuses. (since 4.0) +# Replaces ``locked`` and ``frozen`` statuses. (since 4.0) # # @persistent: true if the bitmap was stored on disk, is scheduled to be stored # on disk, or both. (since 4.0) From patchwork Mon Oct 26 19:42:45 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 301947 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id EA272C4363A for ; Mon, 26 Oct 2020 20:04:13 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 417AA21707 for ; Mon, 26 Oct 2020 20:04:13 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="IPSrFs5U" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 417AA21707 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:50424 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kX8iy-00047P-Ak for qemu-devel@archiver.kernel.org; Mon, 26 Oct 2020 16:04:12 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:33992) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kX8P5-00076K-UW for qemu-devel@nongnu.org; Mon, 26 Oct 2020 15:43:40 -0400 Received: from us-smtp-delivery-124.mimecast.com ([63.128.21.124]:23601) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.90_1) (envelope-from ) id 1kX8P0-0003BP-ND for qemu-devel@nongnu.org; Mon, 26 Oct 2020 15:43:39 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1603741412; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=UaqX6zes8fWQJvF4SbWTPdN5ymaHgrMpHYX+2Td93/I=; b=IPSrFs5URf5zBG7C5Ci7Zn0nIms5EcI3GRYe/oo7EyXAnkTBiyWjJFDUZHozhfGTEJlxtM Z5mO118QWem2ny4zRBz4cKDEDbEn5uK9RWdwIROdm6ChuMDGWlQul3K8lbZhqKIEF2+fMA Ox9Qqo3/RkHUCoGvLZ8bhb13SDz+aJE= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-56-Y5RdwOAlNqqRUiJaOeeLmg-1; Mon, 26 Oct 2020 15:43:17 -0400 X-MC-Unique: Y5RdwOAlNqqRUiJaOeeLmg-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id C7F48192AB97 for ; Mon, 26 Oct 2020 19:43:12 +0000 (UTC) Received: from scv.redhat.com (ovpn-118-238.rdu2.redhat.com [10.10.118.238]) by smtp.corp.redhat.com (Postfix) with ESMTP id 00FC45B4AE; Mon, 26 Oct 2020 19:43:11 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org, Markus Armbruster Subject: [PATCH v2 05/11] qapi/introspect.py: add preliminary type hint annotations Date: Mon, 26 Oct 2020 15:42:45 -0400 Message-Id: <20201026194251.11075-6-jsnow@redhat.com> In-Reply-To: <20201026194251.11075-1-jsnow@redhat.com> References: <20201026194251.11075-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=jsnow@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=63.128.21.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/10/25 21:03:19 X-ACL-Warn: Detected OS = Linux 2.2.x-3.x [generic] [fuzzy] X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: John Snow , Eduardo Habkost , Cleber Rosa Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" The typing of _make_tree and friends is a bit involved, but it can be done with some stubbed out types and a bit of elbow grease. The forthcoming patches attempt to make some simplifications, but having the type hints in advance may aid in review of subsequent patches. Some notes on the abstract types used at this point, and what they represent: - TreeValue represents any object in the type tree. _make_tree is an optional call -- not every node in the final type tree will have been passed to _make_tree, so this type encompasses not only what is passed to _make_tree (dicts, strings) or returned from it (dicts, strings, a 2-tuple), but any recursive value for any of the dicts passed to _make_tree -- which includes lists, strings, integers, null constants, and so on. - _DObject is a type alias I use to mean "A JSON-style object, represented as a Python dict." There is no "JSON" type in Python, they are converted natively to recursively nested dicts and lists, with leaf values of str, int, float, None, True/False and so on. This type structure is not possible to accurately portray in mypy yet, so a placeholder is used. In this case, _DObject is being used to refer to SchemaInfo-like structures as defined in qapi/introspect.json, OR any sub-object values they may reference. We don't have strong typing available for those, so a generic alternative is used. - Extra refers explicitly to the dict containing "extra" information about a node in the tree. mypy does not offer per-key typing for dicts in Python 3.6, so this is the best we can do here. - Annotated refers to (one of) the return types of _make_tree: It represents a 2-tuple of (TreeValue, Extra). Signed-off-by: Eduardo Habkost Signed-off-by: John Snow --- scripts/qapi/introspect.py | 157 ++++++++++++++++++++++++++++--------- scripts/qapi/mypy.ini | 5 -- scripts/qapi/schema.py | 2 +- 3 files changed, 121 insertions(+), 43 deletions(-) diff --git a/scripts/qapi/introspect.py b/scripts/qapi/introspect.py index 63f721ebfb6..803288a64e7 100644 --- a/scripts/qapi/introspect.py +++ b/scripts/qapi/introspect.py @@ -10,7 +10,16 @@ See the COPYING file in the top-level directory. """ -from typing import Optional, Sequence, cast +from typing import ( + Any, + Dict, + List, + Optional, + Sequence, + Tuple, + Union, + cast, +) from .common import ( c_name, @@ -20,13 +29,56 @@ ) from .gen import QAPISchemaMonolithicCVisitor from .schema import ( + QAPISchema, QAPISchemaArrayType, QAPISchemaBuiltinType, + QAPISchemaEntity, + QAPISchemaEnumMember, + QAPISchemaFeature, + QAPISchemaObjectType, + QAPISchemaObjectTypeMember, QAPISchemaType, + QAPISchemaVariant, + QAPISchemaVariants, ) +from .source import QAPISourceInfo -def _make_tree(obj, ifcond, features, extra=None): +# This module constructs a tree-like data structure that is used to +# generate the introspection information for QEMU. It behaves similarly +# to a JSON value. +# +# A complexity over JSON is that our values may or may not be annotated. +# +# Un-annotated values may be: +# Scalar: str, bool, None. +# Non-scalar: List, Dict +# _Value = Union[str, bool, None, Dict[str, Value], List[Value]] +# +# With optional annotations, the type of all values is: +# TreeValue = Union[_Value, Annotated[_Value]] +# +# Sadly, mypy does not support recursive types, so we must approximate this. +_stub = Any +_scalar = Union[str, bool, None] +_nonscalar = Union[Dict[str, _stub], List[_stub]] +_value = Union[_scalar, _nonscalar] +TreeValue = Union[_value, 'Annotated'] + +# This is just an alias for an object in the structure described above: +_DObject = Dict[str, object] + +# Represents the annotations themselves: +Annotations = Dict[str, object] + +# Represents an annotated node (of some kind). +Annotated = Tuple[_value, Annotations] + + +def _make_tree(obj: Union[_DObject, str], ifcond: List[str], + features: List[QAPISchemaFeature], + extra: Optional[Annotations] = None + ) -> TreeValue: if extra is None: extra = {} if ifcond: @@ -39,9 +91,11 @@ def _make_tree(obj, ifcond, features, extra=None): return obj -def _tree_to_qlit(obj, level=0, suppress_first_indent=False): +def _tree_to_qlit(obj: TreeValue, + level: int = 0, + suppress_first_indent: bool = False) -> str: - def indent(level): + def indent(level: int) -> str: return level * 4 * ' ' if isinstance(obj, tuple): @@ -91,21 +145,20 @@ def indent(level): return ret -def to_c_string(string): +def to_c_string(string: str) -> str: return '"' + string.replace('\\', r'\\').replace('"', r'\"') + '"' class QAPISchemaGenIntrospectVisitor(QAPISchemaMonolithicCVisitor): - - def __init__(self, prefix, unmask): + def __init__(self, prefix: str, unmask: bool): super().__init__( prefix, 'qapi-introspect', ' * QAPI/QMP schema introspection', __doc__) self._unmask = unmask - self._schema = None - self._trees = [] - self._used_types = [] - self._name_map = {} + self._schema: Optional[QAPISchema] = None + self._trees: List[TreeValue] = [] + self._used_types: List[QAPISchemaType] = [] + self._name_map: Dict[str, str] = {} self._genc.add(mcgen(''' #include "qemu/osdep.h" #include "%(prefix)sqapi-introspect.h" @@ -113,10 +166,10 @@ def __init__(self, prefix, unmask): ''', prefix=prefix)) - def visit_begin(self, schema): + def visit_begin(self, schema: QAPISchema) -> None: self._schema = schema - def visit_end(self): + def visit_end(self) -> None: # visit the types that are actually used for typ in self._used_types: typ.visit(self) @@ -138,18 +191,18 @@ def visit_end(self): self._used_types = [] self._name_map = {} - def visit_needed(self, entity): + def visit_needed(self, entity: QAPISchemaEntity) -> bool: # Ignore types on first pass; visit_end() will pick up used types return not isinstance(entity, QAPISchemaType) - def _name(self, name): + def _name(self, name: str) -> str: if self._unmask: return name if name not in self._name_map: self._name_map[name] = '%d' % len(self._name_map) return self._name_map[name] - def _use_type(self, typ): + def _use_type(self, typ: QAPISchemaType) -> str: # Map the various integer types to plain int if typ.json_type() == 'int': typ = self._schema.lookup_type('int') @@ -168,8 +221,10 @@ def _use_type(self, typ): return '[' + self._use_type(typ.element_type) + ']' return self._name(typ.name) - def _gen_tree(self, name, mtype, obj, ifcond, features): - extra = None + def _gen_tree(self, name: str, mtype: str, obj: _DObject, + ifcond: List[str], + features: Optional[List[QAPISchemaFeature]]) -> None: + extra: Optional[Annotations] = None if mtype not in ('command', 'event', 'builtin', 'array'): if not self._unmask: # Output a comment to make it easy to map masked names @@ -180,44 +235,64 @@ def _gen_tree(self, name, mtype, obj, ifcond, features): obj['meta-type'] = mtype self._trees.append(_make_tree(obj, ifcond, features, extra)) - def _gen_member(self, member): - obj = {'name': member.name, 'type': self._use_type(member.type)} + def _gen_member(self, + member: QAPISchemaObjectTypeMember) -> TreeValue: + obj: _DObject = { + 'name': member.name, + 'type': self._use_type(member.type) + } if member.optional: obj['default'] = None return _make_tree(obj, member.ifcond, member.features) - def _gen_variants(self, tag_name, variants): + def _gen_variants(self, tag_name: str, + variants: List[QAPISchemaVariant]) -> _DObject: return {'tag': tag_name, 'variants': [self._gen_variant(v) for v in variants]} - def _gen_variant(self, variant): - obj = {'case': variant.name, 'type': self._use_type(variant.type)} + def _gen_variant(self, variant: QAPISchemaVariant) -> TreeValue: + obj: _DObject = { + 'case': variant.name, + 'type': self._use_type(variant.type) + } return _make_tree(obj, variant.ifcond, None) - def visit_builtin_type(self, name, info, json_type): + def visit_builtin_type(self, name: str, info: Optional[QAPISourceInfo], + json_type: str) -> None: self._gen_tree(name, 'builtin', {'json-type': json_type}, [], None) - def visit_enum_type(self, name, info, ifcond, features, members, prefix): + def visit_enum_type(self, name: str, info: QAPISourceInfo, + ifcond: List[str], features: List[QAPISchemaFeature], + members: List[QAPISchemaEnumMember], + prefix: Optional[str]) -> None: self._gen_tree(name, 'enum', {'values': [_make_tree(m.name, m.ifcond, None) for m in members]}, ifcond, features) - def visit_array_type(self, name, info, ifcond, element_type): + def visit_array_type(self, name: str, info: Optional[QAPISourceInfo], + ifcond: List[str], + element_type: QAPISchemaType) -> None: element = self._use_type(element_type) self._gen_tree('[' + element + ']', 'array', {'element-type': element}, ifcond, None) - def visit_object_type_flat(self, name, info, ifcond, features, - members, variants): - obj = {'members': [self._gen_member(m) for m in members]} + def visit_object_type_flat(self, name: str, info: Optional[QAPISourceInfo], + ifcond: List[str], + features: List[QAPISchemaFeature], + members: Sequence[QAPISchemaObjectTypeMember], + variants: Optional[QAPISchemaVariants]) -> None: + obj: _DObject = {'members': [self._gen_member(m) for m in members]} if variants: obj.update(self._gen_variants(variants.tag_member.name, variants.variants)) self._gen_tree(name, 'object', obj, ifcond, features) - def visit_alternate_type(self, name, info, ifcond, features, variants): + def visit_alternate_type(self, name: str, info: QAPISourceInfo, + ifcond: List[str], + features: List[QAPISchemaFeature], + variants: QAPISchemaVariants) -> None: self._gen_tree(name, 'alternate', {'members': [ _make_tree({'type': self._use_type(m.type)}, @@ -225,24 +300,32 @@ def visit_alternate_type(self, name, info, ifcond, features, variants): for m in variants.variants]}, ifcond, features) - def visit_command(self, name, info, ifcond, features, - arg_type, ret_type, gen, success_response, boxed, - allow_oob, allow_preconfig, coroutine): + def visit_command(self, name: str, info: QAPISourceInfo, ifcond: List[str], + features: List[QAPISchemaFeature], + arg_type: QAPISchemaObjectType, + ret_type: Optional[QAPISchemaType], gen: bool, + success_response: bool, boxed: bool, allow_oob: bool, + allow_preconfig: bool, coroutine: bool) -> None: arg_type = arg_type or self._schema.the_empty_object_type ret_type = ret_type or self._schema.the_empty_object_type - obj = {'arg-type': self._use_type(arg_type), - 'ret-type': self._use_type(ret_type)} + obj: _DObject = { + 'arg-type': self._use_type(arg_type), + 'ret-type': self._use_type(ret_type) + } if allow_oob: obj['allow-oob'] = allow_oob self._gen_tree(name, 'command', obj, ifcond, features) - def visit_event(self, name, info, ifcond, features, arg_type, boxed): + def visit_event(self, name: str, info: QAPISourceInfo, + ifcond: List[str], features: List[QAPISchemaFeature], + arg_type: QAPISchemaObjectType, boxed: bool) -> None: arg_type = arg_type or self._schema.the_empty_object_type self._gen_tree(name, 'event', {'arg-type': self._use_type(arg_type)}, ifcond, features) -def gen_introspect(schema, output_dir, prefix, opt_unmask): +def gen_introspect(schema: QAPISchema, output_dir: str, prefix: str, + opt_unmask: bool) -> None: vis = QAPISchemaGenIntrospectVisitor(prefix, opt_unmask) schema.visit(vis) vis.write(output_dir) diff --git a/scripts/qapi/mypy.ini b/scripts/qapi/mypy.ini index 74fc6c82153..c0f2a58306d 100644 --- a/scripts/qapi/mypy.ini +++ b/scripts/qapi/mypy.ini @@ -14,11 +14,6 @@ disallow_untyped_defs = False disallow_incomplete_defs = False check_untyped_defs = False -[mypy-qapi.introspect] -disallow_untyped_defs = False -disallow_incomplete_defs = False -check_untyped_defs = False - [mypy-qapi.parser] disallow_untyped_defs = False disallow_incomplete_defs = False diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py index 720449feee4..e91b77fadc3 100644 --- a/scripts/qapi/schema.py +++ b/scripts/qapi/schema.py @@ -28,7 +28,7 @@ class QAPISchemaEntity: meta: Optional[str] = None - def __init__(self, name, info, doc, ifcond=None, features=None): + def __init__(self, name: str, info, doc, ifcond=None, features=None): assert name is None or isinstance(name, str) for f in features or []: assert isinstance(f, QAPISchemaFeature) From patchwork Mon Oct 26 19:42:46 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 301944 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 41F0DC4363A for ; Mon, 26 Oct 2020 20:10:13 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id DC7CF21BE5 for ; Mon, 26 Oct 2020 20:10:12 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="LVbBkQL0" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org DC7CF21BE5 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:40600 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kX8ol-0003Sb-S3 for qemu-devel@archiver.kernel.org; Mon, 26 Oct 2020 16:10:11 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:33902) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kX8Ow-0006wE-1B for qemu-devel@nongnu.org; Mon, 26 Oct 2020 15:43:30 -0400 Received: from us-smtp-delivery-124.mimecast.com ([63.128.21.124]:36691) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.90_1) (envelope-from ) id 1kX8Ou-0003Ay-13 for qemu-devel@nongnu.org; Mon, 26 Oct 2020 15:43:29 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1603741407; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=7t+TOdJbR+kZy9WPkNJWw2EXuADY2daXwYBT0hG7Sy8=; b=LVbBkQL08nyQqwsxjhKjB2QYmN1WuT5aGnBBPBe6RGZ/FYncCTC0P7WrmZzY5JnRGNEMUn vNGhYXkz9M3RA956NYAnxaKPTJMdf18SHWuZ3fiaHXs7PPbDFX/P+/m0KjC0GDqJGJXg85 kJ+RK9Obnb0FBGdlFfHjLUGQFEp1uDo= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-290-7d82Xk0nPqW5TFPPNtbBUQ-1; Mon, 26 Oct 2020 15:43:25 -0400 X-MC-Unique: 7d82Xk0nPqW5TFPPNtbBUQ-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id B9187AF9DA for ; Mon, 26 Oct 2020 19:43:13 +0000 (UTC) Received: from scv.redhat.com (ovpn-118-238.rdu2.redhat.com [10.10.118.238]) by smtp.corp.redhat.com (Postfix) with ESMTP id EED105B4BA; Mon, 26 Oct 2020 19:43:12 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org, Markus Armbruster Subject: [PATCH v2 06/11] qapi/introspect.py: add _gen_features helper Date: Mon, 26 Oct 2020 15:42:46 -0400 Message-Id: <20201026194251.11075-7-jsnow@redhat.com> In-Reply-To: <20201026194251.11075-1-jsnow@redhat.com> References: <20201026194251.11075-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=jsnow@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=63.128.21.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/10/25 21:03:19 X-ACL-Warn: Detected OS = Linux 2.2.x-3.x [generic] [fuzzy] X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: John Snow , Eduardo Habkost , Cleber Rosa Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" _make_tree might receive a dict or some other type. Adding features information should arguably be performed by the caller at such a time when we know the type of the object and don't have to re-interrogate it. Signed-off-by: John Snow Reviewed-by: Cleber Rosa --- scripts/qapi/introspect.py | 19 ++++++++++++------- 1 file changed, 12 insertions(+), 7 deletions(-) diff --git a/scripts/qapi/introspect.py b/scripts/qapi/introspect.py index 803288a64e7..16282f2634b 100644 --- a/scripts/qapi/introspect.py +++ b/scripts/qapi/introspect.py @@ -76,16 +76,12 @@ def _make_tree(obj: Union[_DObject, str], ifcond: List[str], - features: List[QAPISchemaFeature], extra: Optional[Annotations] = None ) -> TreeValue: if extra is None: extra = {} if ifcond: extra['if'] = ifcond - if features: - assert isinstance(obj, dict) - obj['features'] = [(f.name, {'if': f.ifcond}) for f in features] if extra: return (obj, extra) return obj @@ -221,6 +217,11 @@ def _use_type(self, typ: QAPISchemaType) -> str: return '[' + self._use_type(typ.element_type) + ']' return self._name(typ.name) + @classmethod + def _gen_features(cls, + features: List[QAPISchemaFeature]) -> List[TreeValue]: + return [_make_tree(f.name, f.ifcond) for f in features] + def _gen_tree(self, name: str, mtype: str, obj: _DObject, ifcond: List[str], features: Optional[List[QAPISchemaFeature]]) -> None: @@ -233,7 +234,9 @@ def _gen_tree(self, name: str, mtype: str, obj: _DObject, name = self._name(name) obj['name'] = name obj['meta-type'] = mtype - self._trees.append(_make_tree(obj, ifcond, features, extra)) + if features: + obj['features'] = self._gen_features(features) + self._trees.append(_make_tree(obj, ifcond, extra)) def _gen_member(self, member: QAPISchemaObjectTypeMember) -> TreeValue: @@ -243,7 +246,9 @@ def _gen_member(self, } if member.optional: obj['default'] = None - return _make_tree(obj, member.ifcond, member.features) + if member.features: + obj['features'] = self._gen_features(member.features) + return _make_tree(obj, member.ifcond) def _gen_variants(self, tag_name: str, variants: List[QAPISchemaVariant]) -> _DObject: @@ -255,7 +260,7 @@ def _gen_variant(self, variant: QAPISchemaVariant) -> TreeValue: 'case': variant.name, 'type': self._use_type(variant.type) } - return _make_tree(obj, variant.ifcond, None) + return _make_tree(obj, variant.ifcond) def visit_builtin_type(self, name: str, info: Optional[QAPISourceInfo], json_type: str) -> None: From patchwork Mon Oct 26 19:42:47 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 301941 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 76457C4363A for ; Mon, 26 Oct 2020 20:15:13 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 9E67E21BE5 for ; Mon, 26 Oct 2020 20:15:12 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="idEfhEY7" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 9E67E21BE5 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:54658 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kX8tb-00018Q-JV for qemu-devel@archiver.kernel.org; Mon, 26 Oct 2020 16:15:11 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:33940) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kX8P2-00072a-Op for qemu-devel@nongnu.org; Mon, 26 Oct 2020 15:43:36 -0400 Received: from us-smtp-delivery-124.mimecast.com ([63.128.21.124]:34706) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.90_1) (envelope-from ) id 1kX8P0-0003BJ-MU for qemu-devel@nongnu.org; Mon, 26 Oct 2020 15:43:36 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1603741412; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=REDnd9QKNyIFxZBMsmQu+NVSaxxfmukU0lq6x9MljOQ=; b=idEfhEY78wjGFBxiILlecUV+U8IdzAosZORGRp7T2KKKeLTSNhBu4KP0PNxrClkDwgt0Hl WCG+EfViSDUeO/UAjznJ5nIewBfLYrG2HYYcqLmzEwIDu64s5AQBj4PSGvM1NDOUKV9Arr lSBsRSl5JVOVvj32S9tUP5QguvhQo4k= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-594-2_AaZsLeO56ALMNOEflGag-1; Mon, 26 Oct 2020 15:43:18 -0400 X-MC-Unique: 2_AaZsLeO56ALMNOEflGag-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id A7EFC88EF23 for ; Mon, 26 Oct 2020 19:43:14 +0000 (UTC) Received: from scv.redhat.com (ovpn-118-238.rdu2.redhat.com [10.10.118.238]) by smtp.corp.redhat.com (Postfix) with ESMTP id E1DE85B4B9; Mon, 26 Oct 2020 19:43:13 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org, Markus Armbruster Subject: [PATCH v2 07/11] qapi/introspect.py: Unify return type of _make_tree() Date: Mon, 26 Oct 2020 15:42:47 -0400 Message-Id: <20201026194251.11075-8-jsnow@redhat.com> In-Reply-To: <20201026194251.11075-1-jsnow@redhat.com> References: <20201026194251.11075-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=jsnow@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=63.128.21.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/10/25 21:03:19 X-ACL-Warn: Detected OS = Linux 2.2.x-3.x [generic] [fuzzy] X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: John Snow , Eduardo Habkost , Cleber Rosa Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Returning two different types conditionally can be complicated to type. Let's always return a tuple for consistency. Prohibit the use of annotations with dict-values in this circumstance. It can be implemented later if and when the need for it arises. Signed-off-by: John Snow --- scripts/qapi/introspect.py | 21 ++++++++++++--------- 1 file changed, 12 insertions(+), 9 deletions(-) diff --git a/scripts/qapi/introspect.py b/scripts/qapi/introspect.py index 16282f2634b..ef469b6c06e 100644 --- a/scripts/qapi/introspect.py +++ b/scripts/qapi/introspect.py @@ -77,14 +77,12 @@ def _make_tree(obj: Union[_DObject, str], ifcond: List[str], extra: Optional[Annotations] = None - ) -> TreeValue: + ) -> Annotated: if extra is None: extra = {} if ifcond: extra['if'] = ifcond - if extra: - return (obj, extra) - return obj + return (obj, extra) def _tree_to_qlit(obj: TreeValue, @@ -98,12 +96,16 @@ def indent(level: int) -> str: ifobj, extra = obj ifcond = cast(Optional[Sequence[str]], extra.get('if')) comment = extra.get('comment') + + msg = "Comments and Conditionals not implemented for dict values" + assert not (suppress_first_indent and (ifcond or comment)), msg + ret = '' if comment: ret += indent(level) + '/* %s */\n' % comment if ifcond: ret += gen_if(ifcond) - ret += _tree_to_qlit(ifobj, level) + ret += _tree_to_qlit(ifobj, level, suppress_first_indent) if ifcond: ret += '\n' + gen_endif(ifcond) return ret @@ -152,7 +154,7 @@ def __init__(self, prefix: str, unmask: bool): ' * QAPI/QMP schema introspection', __doc__) self._unmask = unmask self._schema: Optional[QAPISchema] = None - self._trees: List[TreeValue] = [] + self._trees: List[Annotated] = [] self._used_types: List[QAPISchemaType] = [] self._name_map: Dict[str, str] = {} self._genc.add(mcgen(''' @@ -219,7 +221,8 @@ def _use_type(self, typ: QAPISchemaType) -> str: @classmethod def _gen_features(cls, - features: List[QAPISchemaFeature]) -> List[TreeValue]: + features: List[QAPISchemaFeature] + ) -> List[Annotated]: return [_make_tree(f.name, f.ifcond) for f in features] def _gen_tree(self, name: str, mtype: str, obj: _DObject, @@ -239,7 +242,7 @@ def _gen_tree(self, name: str, mtype: str, obj: _DObject, self._trees.append(_make_tree(obj, ifcond, extra)) def _gen_member(self, - member: QAPISchemaObjectTypeMember) -> TreeValue: + member: QAPISchemaObjectTypeMember) -> Annotated: obj: _DObject = { 'name': member.name, 'type': self._use_type(member.type) @@ -255,7 +258,7 @@ def _gen_variants(self, tag_name: str, return {'tag': tag_name, 'variants': [self._gen_variant(v) for v in variants]} - def _gen_variant(self, variant: QAPISchemaVariant) -> TreeValue: + def _gen_variant(self, variant: QAPISchemaVariant) -> Annotated: obj: _DObject = { 'case': variant.name, 'type': self._use_type(variant.type) From patchwork Mon Oct 26 19:42:49 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 301948 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E1079C4363A for ; Mon, 26 Oct 2020 20:00:53 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 44A7420795 for ; Mon, 26 Oct 2020 20:00:53 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="hAtnWDSS" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 44A7420795 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:43348 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kX8fk-0000vk-2d for qemu-devel@archiver.kernel.org; Mon, 26 Oct 2020 16:00:52 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:33990) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kX8P5-000760-Da for qemu-devel@nongnu.org; Mon, 26 Oct 2020 15:43:39 -0400 Received: from us-smtp-delivery-124.mimecast.com ([63.128.21.124]:51832) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.90_1) (envelope-from ) id 1kX8P3-0003Bl-2t for qemu-devel@nongnu.org; Mon, 26 Oct 2020 15:43:39 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1603741416; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=92mglJF7EP28EAbroH+YcEyPHAhA2QfdRodb5cb2HGU=; b=hAtnWDSSqU6PpSi75o8umSWl3w9G2ogIR9DF7q9yPJdJl3zCl1Clok0Xbce766xZGCRYz+ /Q6MZaulr12+Nrtv0ntozaiQ89qqth0duwwH+c5BMQuEXgzPd9kDHDfu5ij6p5HTR6bUod IWMNw/4ajVaLMHS6DDE/F/q4GeTOo3A= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-594-yR8N5U-pM6Cpvwhf3cQGnQ-1; Mon, 26 Oct 2020 15:43:29 -0400 X-MC-Unique: yR8N5U-pM6Cpvwhf3cQGnQ-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 8EF4FAE572 for ; Mon, 26 Oct 2020 19:43:16 +0000 (UTC) Received: from scv.redhat.com (ovpn-118-238.rdu2.redhat.com [10.10.118.238]) by smtp.corp.redhat.com (Postfix) with ESMTP id CAC545B4AE; Mon, 26 Oct 2020 19:43:15 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org, Markus Armbruster Subject: [PATCH v2 09/11] qapi/introspect.py: create a typed 'Annotated' data strutcure Date: Mon, 26 Oct 2020 15:42:49 -0400 Message-Id: <20201026194251.11075-10-jsnow@redhat.com> In-Reply-To: <20201026194251.11075-1-jsnow@redhat.com> References: <20201026194251.11075-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=jsnow@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=63.128.21.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/10/25 21:03:19 X-ACL-Warn: Detected OS = Linux 2.2.x-3.x [generic] [fuzzy] X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: John Snow , Eduardo Habkost , Cleber Rosa Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" This replaces _make_tree with Annotated(). By creating it as a generic container, we can more accurately describe the exact nature of this particular value. i.e., each Annotated object is actually an Annotated, describing its contained value. This adds stricter typing to Annotated nodes and extra annotated information. It also replaces a check of "isinstance tuple" with the much more explicit "isinstance Annotated" which is guaranteed not to break if a tuple is accidentally introduced into the type tree. (Perhaps as a result of a bad conversion from a list.) Signed-off-by: John Snow Reviewed-by: Cleber Rosa --- scripts/qapi/introspect.py | 97 +++++++++++++++++++------------------- 1 file changed, 48 insertions(+), 49 deletions(-) diff --git a/scripts/qapi/introspect.py b/scripts/qapi/introspect.py index a0978cb3adb..a261e402d69 100644 --- a/scripts/qapi/introspect.py +++ b/scripts/qapi/introspect.py @@ -13,12 +13,13 @@ from typing import ( Any, Dict, + Generic, + Iterable, List, Optional, Sequence, - Tuple, + TypeVar, Union, - cast, ) from .common import ( @@ -63,50 +64,48 @@ _scalar = Union[str, bool, None] _nonscalar = Union[Dict[str, _stub], List[_stub]] _value = Union[_scalar, _nonscalar] -TreeValue = Union[_value, 'Annotated'] +TreeValue = Union[_value, 'Annotated[_value]'] # This is just an alias for an object in the structure described above: _DObject = Dict[str, object] -# Represents the annotations themselves: -Annotations = Dict[str, object] -# Represents an annotated node (of some kind). -Annotated = Tuple[_value, Annotations] +_AnnoType = TypeVar('_AnnoType', bound=TreeValue) -def _make_tree(obj: Union[_DObject, str], ifcond: List[str], - comment: Optional[str] = None) -> Annotated: - extra: Annotations = { - 'if': ifcond, - 'comment': comment, - } - return (obj, extra) +class Annotated(Generic[_AnnoType]): + """ + Annotated generally contains a SchemaInfo-like type (as a dict), + But it also used to wrap comments/ifconds around scalar leaf values, + for the benefit of features and enums. + """ + # Remove after 3.7 adds @dataclass: + # pylint: disable=too-few-public-methods + def __init__(self, value: _AnnoType, ifcond: Iterable[str], + comment: Optional[str] = None): + self.value = value + self.comment: Optional[str] = comment + self.ifcond: Sequence[str] = tuple(ifcond) -def _tree_to_qlit(obj: TreeValue, - level: int = 0, +def _tree_to_qlit(obj: TreeValue, level: int = 0, suppress_first_indent: bool = False) -> str: def indent(level: int) -> str: return level * 4 * ' ' - if isinstance(obj, tuple): - ifobj, extra = obj - ifcond = cast(Optional[Sequence[str]], extra.get('if')) - comment = extra.get('comment') - + if isinstance(obj, Annotated): msg = "Comments and Conditionals not implemented for dict values" - assert not (suppress_first_indent and (ifcond or comment)), msg + assert not (suppress_first_indent and (obj.comment or obj.ifcond)), msg ret = '' - if comment: - ret += indent(level) + '/* %s */\n' % comment - if ifcond: - ret += gen_if(ifcond) - ret += _tree_to_qlit(ifobj, level, suppress_first_indent) - if ifcond: - ret += '\n' + gen_endif(ifcond) + if obj.comment: + ret += indent(level) + '/* %s */\n' % obj.comment + if obj.ifcond: + ret += gen_if(obj.ifcond) + ret += _tree_to_qlit(obj.value, level, suppress_first_indent) + if obj.ifcond: + ret += '\n' + gen_endif(obj.ifcond) return ret ret = '' @@ -153,7 +152,7 @@ def __init__(self, prefix: str, unmask: bool): ' * QAPI/QMP schema introspection', __doc__) self._unmask = unmask self._schema: Optional[QAPISchema] = None - self._trees: List[Annotated] = [] + self._trees: List[Annotated[_DObject]] = [] self._used_types: List[QAPISchemaType] = [] self._name_map: Dict[str, str] = {} self._genc.add(mcgen(''' @@ -219,10 +218,9 @@ def _use_type(self, typ: QAPISchemaType) -> str: return self._name(typ.name) @classmethod - def _gen_features(cls, - features: List[QAPISchemaFeature] - ) -> List[Annotated]: - return [_make_tree(f.name, f.ifcond) for f in features] + def _gen_features( + cls, features: List[QAPISchemaFeature]) -> List[Annotated[str]]: + return [Annotated(f.name, f.ifcond) for f in features] def _gen_tree(self, name: str, mtype: str, obj: _DObject, ifcond: List[str], @@ -238,10 +236,10 @@ def _gen_tree(self, name: str, mtype: str, obj: _DObject, obj['meta-type'] = mtype if features: obj['features'] = self._gen_features(features) - self._trees.append(_make_tree(obj, ifcond, comment)) + self._trees.append(Annotated(obj, ifcond, comment)) def _gen_member(self, - member: QAPISchemaObjectTypeMember) -> Annotated: + member: QAPISchemaObjectTypeMember) -> Annotated[_DObject]: obj: _DObject = { 'name': member.name, 'type': self._use_type(member.type) @@ -250,19 +248,19 @@ def _gen_member(self, obj['default'] = None if member.features: obj['features'] = self._gen_features(member.features) - return _make_tree(obj, member.ifcond) + return Annotated(obj, member.ifcond) def _gen_variants(self, tag_name: str, variants: List[QAPISchemaVariant]) -> _DObject: return {'tag': tag_name, 'variants': [self._gen_variant(v) for v in variants]} - def _gen_variant(self, variant: QAPISchemaVariant) -> Annotated: + def _gen_variant(self, variant: QAPISchemaVariant) -> Annotated[_DObject]: obj: _DObject = { 'case': variant.name, 'type': self._use_type(variant.type) } - return _make_tree(obj, variant.ifcond) + return Annotated(obj, variant.ifcond) def visit_builtin_type(self, name: str, info: Optional[QAPISourceInfo], json_type: str) -> None: @@ -272,10 +270,11 @@ def visit_enum_type(self, name: str, info: QAPISourceInfo, ifcond: List[str], features: List[QAPISchemaFeature], members: List[QAPISchemaEnumMember], prefix: Optional[str]) -> None: - self._gen_tree(name, 'enum', - {'values': [_make_tree(m.name, m.ifcond, None) - for m in members]}, - ifcond, features) + self._gen_tree( + name, 'enum', + {'values': [Annotated(m.name, m.ifcond) for m in members]}, + ifcond, features + ) def visit_array_type(self, name: str, info: Optional[QAPISourceInfo], ifcond: List[str], @@ -300,12 +299,12 @@ def visit_alternate_type(self, name: str, info: QAPISourceInfo, ifcond: List[str], features: List[QAPISchemaFeature], variants: QAPISchemaVariants) -> None: - self._gen_tree(name, 'alternate', - {'members': [ - _make_tree({'type': self._use_type(m.type)}, - m.ifcond, None) - for m in variants.variants]}, - ifcond, features) + self._gen_tree( + name, 'alternate', + {'members': [Annotated({'type': self._use_type(m.type)}, m.ifcond) + for m in variants.variants]}, + ifcond, features + ) def visit_command(self, name: str, info: QAPISourceInfo, ifcond: List[str], features: List[QAPISchemaFeature], From patchwork Mon Oct 26 19:42:50 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 301942 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C730FC4363A for ; Mon, 26 Oct 2020 20:13:09 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 4685921BE5 for ; Mon, 26 Oct 2020 20:13:09 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="VlcDo2hY" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4685921BE5 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:48762 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kX8rc-00070d-74 for qemu-devel@archiver.kernel.org; Mon, 26 Oct 2020 16:13:08 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:33986) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kX8P4-00075Q-Pd for qemu-devel@nongnu.org; Mon, 26 Oct 2020 15:43:39 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:22086) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.90_1) (envelope-from ) id 1kX8P0-0003BT-Nz for qemu-devel@nongnu.org; Mon, 26 Oct 2020 15:43:38 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1603741413; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=on5YqcSIhfWljRLv4oDXCK2tq8+YrnQBZVCFBdMkehY=; b=VlcDo2hYebsVfNQ9yZMu2JE3JcXWdj935P7POTu8b/c76yAcKZKPnWA2xVpE3rnl12Tt62 qyZ52qGRfnuFbkS5mk73e6vpSe/nQLdBh8jnEnToFHDE1Imm/vADZKtJ2Ysj/gfuDahQc9 HamP/UpV8hW9DZ13XDp4titItUjINIQ= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-411-RFifaAU8MNeEnWIOAEavlQ-1; Mon, 26 Oct 2020 15:43:30 -0400 X-MC-Unique: RFifaAU8MNeEnWIOAEavlQ-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id C319A1084CA3 for ; Mon, 26 Oct 2020 19:43:17 +0000 (UTC) Received: from scv.redhat.com (ovpn-118-238.rdu2.redhat.com [10.10.118.238]) by smtp.corp.redhat.com (Postfix) with ESMTP id B67A25B4AE; Mon, 26 Oct 2020 19:43:16 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org, Markus Armbruster Subject: [PATCH v2 10/11] qapi/introspect.py: improve readability of _tree_to_qlit Date: Mon, 26 Oct 2020 15:42:50 -0400 Message-Id: <20201026194251.11075-11-jsnow@redhat.com> In-Reply-To: <20201026194251.11075-1-jsnow@redhat.com> References: <20201026194251.11075-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=jsnow@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=216.205.24.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/10/26 02:39:09 X-ACL-Warn: Detected OS = Linux 2.2.x-3.x [generic] [fuzzy] X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: John Snow , Eduardo Habkost , Cleber Rosa Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Subjective, but I find getting rid of the comprehensions helps. Also, divide the sections into scalar and non-scalar sections, and remove old-style string formatting. Signed-off-by: John Snow --- scripts/qapi/introspect.py | 37 +++++++++++++++++++++---------------- 1 file changed, 21 insertions(+), 16 deletions(-) diff --git a/scripts/qapi/introspect.py b/scripts/qapi/introspect.py index a261e402d69..d4f28485ba5 100644 --- a/scripts/qapi/introspect.py +++ b/scripts/qapi/introspect.py @@ -100,7 +100,7 @@ def indent(level: int) -> str: ret = '' if obj.comment: - ret += indent(level) + '/* %s */\n' % obj.comment + ret += indent(level) + f"/* {obj.comment} */\n" if obj.ifcond: ret += gen_if(obj.ifcond) ret += _tree_to_qlit(obj.value, level, suppress_first_indent) @@ -111,31 +111,36 @@ def indent(level: int) -> str: ret = '' if not suppress_first_indent: ret += indent(level) + + # Scalars: if obj is None: ret += 'QLIT_QNULL' elif isinstance(obj, str): - ret += 'QLIT_QSTR(' + to_c_string(obj) + ')' + ret += f"QLIT_QSTR({to_c_string(obj)})" + elif isinstance(obj, bool): + ret += "QLIT_QBOOL({:s})".format(str(obj).lower()) + + # Non-scalars: elif isinstance(obj, list): - elts = [_tree_to_qlit(elt, level + 1).strip('\n') - for elt in obj] - elts.append(indent(level + 1) + "{}") ret += 'QLIT_QLIST(((QLitObject[]) {\n' - ret += '\n'.join(elts) + '\n' + for value in obj: + ret += _tree_to_qlit(value, level + 1).strip('\n') + '\n' + ret += indent(level + 1) + '{}\n' ret += indent(level) + '}))' elif isinstance(obj, dict): - elts = [] - for key, value in sorted(obj.items()): - elts.append(indent(level + 1) + '{ %s, %s }' % - (to_c_string(key), - _tree_to_qlit(value, level + 1, True))) - elts.append(indent(level + 1) + '{}') ret += 'QLIT_QDICT(((QLitDictEntry[]) {\n' - ret += ',\n'.join(elts) + '\n' + for key, value in sorted(obj.items()): + ret += indent(level + 1) + "{{ {:s}, {:s} }},\n".format( + to_c_string(key), + _tree_to_qlit(value, level + 1, suppress_first_indent=True) + ) + ret += indent(level + 1) + '{}\n' ret += indent(level) + '}))' - elif isinstance(obj, bool): - ret += 'QLIT_QBOOL(%s)' % ('true' if obj else 'false') else: - assert False # not implemented + raise NotImplementedError( + f"type '{type(obj).__name__}' not implemented" + ) + if level > 0: ret += ',' return ret From patchwork Mon Oct 26 19:42:51 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 301939 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3A364C4363A for ; Mon, 26 Oct 2020 20:24:43 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id B030621D41 for ; Mon, 26 Oct 2020 20:24:42 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="Lekr2kBX" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B030621D41 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:48574 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kX92n-0002OV-IR for qemu-devel@archiver.kernel.org; Mon, 26 Oct 2020 16:24:41 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:33880) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kX8Os-0006op-Pf for qemu-devel@nongnu.org; Mon, 26 Oct 2020 15:43:26 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:38289) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.90_1) (envelope-from ) id 1kX8Or-0003AU-2x for qemu-devel@nongnu.org; Mon, 26 Oct 2020 15:43:26 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1603741404; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=w3CcewjopndRGh33OAF+Hw1ML2kb8rBcP4jGOP9my7Q=; b=Lekr2kBXPcGO40gE3NDsNKamWhYuxDLU8pi+gdoaXGawz6ZFpQZDAxUzS9a5EklVOaogwO +GWvC6Ypdtg+ixzrAJZjZuLeq5XX52XqAV2Jq9b5DOw93MIojnF5/R/AwO0V/PdhNIq+ib SpZw+I/2pvQAONv45DZ4INT7D6B3po8= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-235-wM1BPADTPSSn-PBp5HKTJg-1; Mon, 26 Oct 2020 15:43:22 -0400 X-MC-Unique: wM1BPADTPSSn-PBp5HKTJg-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id D3733195D575 for ; Mon, 26 Oct 2020 19:43:18 +0000 (UTC) Received: from scv.redhat.com (ovpn-118-238.rdu2.redhat.com [10.10.118.238]) by smtp.corp.redhat.com (Postfix) with ESMTP id EA0E05B4AE; Mon, 26 Oct 2020 19:43:17 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org, Markus Armbruster Subject: [PATCH v2 11/11] qapi/introspect.py: Add docstring to _tree_to_qlit Date: Mon, 26 Oct 2020 15:42:51 -0400 Message-Id: <20201026194251.11075-12-jsnow@redhat.com> In-Reply-To: <20201026194251.11075-1-jsnow@redhat.com> References: <20201026194251.11075-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=jsnow@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=216.205.24.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/10/26 02:39:09 X-ACL-Warn: Detected OS = Linux 2.2.x-3.x [generic] [fuzzy] X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: John Snow , Eduardo Habkost , Cleber Rosa Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Signed-off-by: John Snow Reviewed-by: Cleber Rosa --- scripts/qapi/introspect.py | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/scripts/qapi/introspect.py b/scripts/qapi/introspect.py index d4f28485ba5..8db9650adbc 100644 --- a/scripts/qapi/introspect.py +++ b/scripts/qapi/introspect.py @@ -1,10 +1,11 @@ """ QAPI introspection generator -Copyright (C) 2015-2018 Red Hat, Inc. +Copyright (C) 2015-2020 Red Hat, Inc. Authors: Markus Armbruster + John Snow This work is licensed under the terms of the GNU GPL, version 2. See the COPYING file in the top-level directory. @@ -90,6 +91,13 @@ def __init__(self, value: _AnnoType, ifcond: Iterable[str], def _tree_to_qlit(obj: TreeValue, level: int = 0, suppress_first_indent: bool = False) -> str: + """ + Convert the type tree into a QLIT C string, recursively. + + :param obj: The value to convert. + :param level: The indentation level for this particular value. + :param suppress_first_indent: True for dict value children. + """ def indent(level: int) -> str: return level * 4 * ' '