mbox series

[v3,00/15] python: create installable package

Message ID 20201020193555.1493936-1-jsnow@redhat.com
Headers show
Series python: create installable package | expand

Message

John Snow Oct. 20, 2020, 7:35 p.m. UTC
Based-on: https://gitlab.com/jsnow/qemu/-/tree/python

This series factors the python/qemu directory as an installable
package. It does not yet actually change the mechanics of how any other
python source in the tree actually consumes it (yet), beyond the import
path.

The point of this series is primarily to formalize our dependencies on
mypy, flake8, isort, and pylint alongside versions that are known to
work. It also adds explicitly pinned versions of these dependencies that
should behave in a repeatable and known way for developers and CI
environments both.

With the python tooling as a proper package, you can install this
package in editable or production mode to a virtual environment, your
local user environment, or your system packages. The primary benefit of
this is to gain access to QMP tooling regardless of CWD, without needing
to battle sys.path.

For example: when developing, you may go to qemu/python/ and invoke
`pipenv shell` to activate a virtual environment that contains the qemu
packages.  This package will always reflect the current version of the
source files in the tree. When you are finished, you can simply exit the
shell to remove these packages from your python environment.

When not developing, you could install a version of this package to your
environment outright to gain access to the QMP and QEMUMachine classes
for lightweight scripting and testing by using pip: "pip install [--user] ."

Finally, this package is formatted in such a way that it COULD be
uploaded to https://pypi.org/project/qemu and installed independently of
qemu.git with `pip install qemu`, but that button remains unpushed.

TESTING THIS SERIES:

CD to qemu/python first, and then:

1. Try "pipenv shell" to get a venv with the package installed to it in
editable mode. Ctrl+d exits this venv shell. While in this shell, any
python script that uses "from qemu.[qmp|machine] import ..." should work
correctly regardless of your CWD.

2. Try "pipenv sync --dev" to create/update the venv with the
development packages without actually entering the venv. This should
install isort, mypy, flake8 and pylint to the venv.

3. After the above sync, try "pipenv shell" again, and from the python
project root, try any of the following:

  - pylint qemu
  - flake8 qemu
  - isort -c qemu
  - mypy qemu

4. Leave any venv you are in, and from the project root, try the
following commands:

  - pipenv run pylint qemu
  - pipenv run flake8 qemu
  - pipenv run isort -c qemu
  - pipenv run mypy qemu

V3:
 - Changed "qemu.core" to "qemu.qmp" and "qemu.machine",
   Partly to accommodate forthcoming work which would benefit from a separate
   qemu.qmp namespace.
 - Changed the initial version from 5.2.0a1 to 0.5.2.0a1, to allow for
   more rapid development while we smooth out the initial kinks.
 - 001: Renamed patch title; differences implement the new names.
 - 002: Readme changes for above.
 - 003: Version change.
 - 004: New readme for the new qmp directory.
 - 006: A few more import exceptions for pylint, hopefully temporary.
 - 009: Updated flake8 config comment to match qapi's
 - 012: Added namespace_package configuration value

001/15:[down] 'python: create qemu packages'
002/15:[0009] [FC] 'python: add qemu package installer'
003/15:[0002] [FC] 'python: add VERSION file'
004/15:[0015] [FC] 'python: add directory structure README.rst files'
005/15:[----] [--] 'python: Add pipenv support'
006/15:[down] 'python: add pylint import exceptions'
007/15:[----] [--] 'python: move pylintrc into setup.cfg'
008/15:[----] [--] 'python: add pylint to pipenv'
009/15:[0002] [FC] 'python: move flake8 config to setup.cfg'
010/15:[----] [--] 'python: Add flake8 to pipenv'
011/15:[----] [-C] 'python: move mypy.ini into setup.cfg'
012/15:[0001] [FC] 'python: add mypy to pipenv'
013/15:[----] [--] 'python: move .isort.cfg into setup.cfg'
014/15:[----] [--] 'python/qemu: add isort to pipenv'
015/15:[----] [--] 'python/qemu: add qemu package itself to pipenv'

John Snow (15):
  python: create qemu packages
  python: add qemu package installer
  python: add VERSION file
  python: add directory structure README.rst files
  python: Add pipenv support
  python: add pylint import exceptions
  python: move pylintrc into setup.cfg
  python: add pylint to pipenv
  python: move flake8 config to setup.cfg
  python: Add flake8 to pipenv
  python: move mypy.ini into setup.cfg
  python: add mypy to pipenv
  python: move .isort.cfg into setup.cfg
  python/qemu: add isort to pipenv
  python/qemu: add qemu package itself to pipenv

 python/PACKAGE.rst                          |  26 +++
 python/README.rst                           |  27 +++
 python/qemu/README.rst                      |   8 +
 python/qemu/machine/README.rst              |   9 +
 python/qemu/qmp/README.rst                  |   9 +
 python/Pipfile                              |  16 ++
 python/Pipfile.lock                         | 207 ++++++++++++++++++++
 python/VERSION                              |   1 +
 python/mypy.ini                             |   4 -
 python/qemu/.flake8                         |   2 -
 python/qemu/.isort.cfg                      |   7 -
 python/qemu/__init__.py                     |  11 --
 python/qemu/machine/__init__.py             |  44 +++++
 python/qemu/{ => machine}/accel.py          |   0
 python/qemu/{ => machine}/console_socket.py |   0
 python/qemu/{ => machine}/machine.py        |  16 +-
 python/qemu/{ => machine}/qtest.py          |   3 +-
 python/qemu/{qmp.py => qmp/__init__.py}     |  12 +-
 python/{qemu/pylintrc => setup.cfg}         |  67 ++++---
 python/setup.py                             |  23 +++
 tests/acceptance/boot_linux.py              |   3 +-
 tests/qemu-iotests/297                      |   2 +-
 tests/qemu-iotests/300                      |   4 +-
 tests/qemu-iotests/iotests.py               |   2 +-
 tests/vm/basevm.py                          |   3 +-
 25 files changed, 437 insertions(+), 69 deletions(-)
 create mode 100644 python/PACKAGE.rst
 create mode 100644 python/README.rst
 create mode 100644 python/qemu/README.rst
 create mode 100644 python/qemu/machine/README.rst
 create mode 100644 python/qemu/qmp/README.rst
 create mode 100644 python/Pipfile
 create mode 100644 python/Pipfile.lock
 create mode 100644 python/VERSION
 delete mode 100644 python/mypy.ini
 delete mode 100644 python/qemu/.flake8
 delete mode 100644 python/qemu/.isort.cfg
 delete mode 100644 python/qemu/__init__.py
 create mode 100644 python/qemu/machine/__init__.py
 rename python/qemu/{ => machine}/accel.py (100%)
 rename python/qemu/{ => machine}/console_socket.py (100%)
 rename python/qemu/{ => machine}/machine.py (98%)
 rename python/qemu/{ => machine}/qtest.py (98%)
 rename python/qemu/{qmp.py => qmp/__init__.py} (96%)
 rename python/{qemu/pylintrc => setup.cfg} (50%)
 mode change 100644 => 100755
 create mode 100755 python/setup.py

-- 
2.26.2

Comments

John Snow Oct. 27, 2020, 10:08 p.m. UTC | #1
Ping O:-)

Looking for feedback from at least Cleber and Eduardo before I barge 
ahead and send a PR to include this on master. Additional packaging and 
versioning feedback from Dan would be nice.

(I know we have a very busy two weeks here; I will continue pinging, but 
I have every intention of merging this prior to 5.2.)

--js

On 10/20/20 3:35 PM, John Snow wrote:
> Based-on: https://gitlab.com/jsnow/qemu/-/tree/python
> 
> This series factors the python/qemu directory as an installable
> package. It does not yet actually change the mechanics of how any other
> python source in the tree actually consumes it (yet), beyond the import
> path.
>  > The point of this series is primarily to formalize our dependencies on
> mypy, flake8, isort, and pylint alongside versions that are known to
> work. It also adds explicitly pinned versions of these dependencies that
> should behave in a repeatable and known way for developers and CI
> environments both.
> 
> With the python tooling as a proper package, you can install this
> package in editable or production mode to a virtual environment, your
> local user environment, or your system packages. The primary benefit of
> this is to gain access to QMP tooling regardless of CWD, without needing
> to battle sys.path.
> 
> For example: when developing, you may go to qemu/python/ and invoke
> `pipenv shell` to activate a virtual environment that contains the qemu
> packages.  This package will always reflect the current version of the
> source files in the tree. When you are finished, you can simply exit the
> shell to remove these packages from your python environment.
> 
> When not developing, you could install a version of this package to your
> environment outright to gain access to the QMP and QEMUMachine classes
> for lightweight scripting and testing by using pip: "pip install [--user] ."
> 
> Finally, this package is formatted in such a way that it COULD be
> uploaded to https://pypi.org/project/qemu and installed independently of
> qemu.git with `pip install qemu`, but that button remains unpushed.
> 
> TESTING THIS SERIES:
> 
> CD to qemu/python first, and then:
> 
> 1. Try "pipenv shell" to get a venv with the package installed to it in
> editable mode. Ctrl+d exits this venv shell. While in this shell, any
> python script that uses "from qemu.[qmp|machine] import ..." should work
> correctly regardless of your CWD.
> 
> 2. Try "pipenv sync --dev" to create/update the venv with the
> development packages without actually entering the venv. This should
> install isort, mypy, flake8 and pylint to the venv.
> 
> 3. After the above sync, try "pipenv shell" again, and from the python
> project root, try any of the following:
> 
>    - pylint qemu
>    - flake8 qemu
>    - isort -c qemu
>    - mypy qemu
> 
> 4. Leave any venv you are in, and from the project root, try the
> following commands:
> 
>    - pipenv run pylint qemu
>    - pipenv run flake8 qemu
>    - pipenv run isort -c qemu
>    - pipenv run mypy qemu
> 
> V3:
>   - Changed "qemu.core" to "qemu.qmp" and "qemu.machine",
>     Partly to accommodate forthcoming work which would benefit from a separate
>     qemu.qmp namespace.
>   - Changed the initial version from 5.2.0a1 to 0.5.2.0a1, to allow for
>     more rapid development while we smooth out the initial kinks.
>   - 001: Renamed patch title; differences implement the new names.
>   - 002: Readme changes for above.
>   - 003: Version change.
>   - 004: New readme for the new qmp directory.
>   - 006: A few more import exceptions for pylint, hopefully temporary.
>   - 009: Updated flake8 config comment to match qapi's
>   - 012: Added namespace_package configuration value
> 
> 001/15:[down] 'python: create qemu packages'
> 002/15:[0009] [FC] 'python: add qemu package installer'
> 003/15:[0002] [FC] 'python: add VERSION file'
> 004/15:[0015] [FC] 'python: add directory structure README.rst files'
> 005/15:[----] [--] 'python: Add pipenv support'
> 006/15:[down] 'python: add pylint import exceptions'
> 007/15:[----] [--] 'python: move pylintrc into setup.cfg'
> 008/15:[----] [--] 'python: add pylint to pipenv'
> 009/15:[0002] [FC] 'python: move flake8 config to setup.cfg'
> 010/15:[----] [--] 'python: Add flake8 to pipenv'
> 011/15:[----] [-C] 'python: move mypy.ini into setup.cfg'
> 012/15:[0001] [FC] 'python: add mypy to pipenv'
> 013/15:[----] [--] 'python: move .isort.cfg into setup.cfg'
> 014/15:[----] [--] 'python/qemu: add isort to pipenv'
> 015/15:[----] [--] 'python/qemu: add qemu package itself to pipenv'
> 
> John Snow (15):
>    python: create qemu packages
>    python: add qemu package installer
>    python: add VERSION file
>    python: add directory structure README.rst files
>    python: Add pipenv support
>    python: add pylint import exceptions
>    python: move pylintrc into setup.cfg
>    python: add pylint to pipenv
>    python: move flake8 config to setup.cfg
>    python: Add flake8 to pipenv
>    python: move mypy.ini into setup.cfg
>    python: add mypy to pipenv
>    python: move .isort.cfg into setup.cfg
>    python/qemu: add isort to pipenv
>    python/qemu: add qemu package itself to pipenv
> 
>   python/PACKAGE.rst                          |  26 +++
>   python/README.rst                           |  27 +++
>   python/qemu/README.rst                      |   8 +
>   python/qemu/machine/README.rst              |   9 +
>   python/qemu/qmp/README.rst                  |   9 +
>   python/Pipfile                              |  16 ++
>   python/Pipfile.lock                         | 207 ++++++++++++++++++++
>   python/VERSION                              |   1 +
>   python/mypy.ini                             |   4 -
>   python/qemu/.flake8                         |   2 -
>   python/qemu/.isort.cfg                      |   7 -
>   python/qemu/__init__.py                     |  11 --
>   python/qemu/machine/__init__.py             |  44 +++++
>   python/qemu/{ => machine}/accel.py          |   0
>   python/qemu/{ => machine}/console_socket.py |   0
>   python/qemu/{ => machine}/machine.py        |  16 +-
>   python/qemu/{ => machine}/qtest.py          |   3 +-
>   python/qemu/{qmp.py => qmp/__init__.py}     |  12 +-
>   python/{qemu/pylintrc => setup.cfg}         |  67 ++++---
>   python/setup.py                             |  23 +++
>   tests/acceptance/boot_linux.py              |   3 +-
>   tests/qemu-iotests/297                      |   2 +-
>   tests/qemu-iotests/300                      |   4 +-
>   tests/qemu-iotests/iotests.py               |   2 +-
>   tests/vm/basevm.py                          |   3 +-
>   25 files changed, 437 insertions(+), 69 deletions(-)
>   create mode 100644 python/PACKAGE.rst
>   create mode 100644 python/README.rst
>   create mode 100644 python/qemu/README.rst
>   create mode 100644 python/qemu/machine/README.rst
>   create mode 100644 python/qemu/qmp/README.rst
>   create mode 100644 python/Pipfile
>   create mode 100644 python/Pipfile.lock
>   create mode 100644 python/VERSION
>   delete mode 100644 python/mypy.ini
>   delete mode 100644 python/qemu/.flake8
>   delete mode 100644 python/qemu/.isort.cfg
>   delete mode 100644 python/qemu/__init__.py
>   create mode 100644 python/qemu/machine/__init__.py
>   rename python/qemu/{ => machine}/accel.py (100%)
>   rename python/qemu/{ => machine}/console_socket.py (100%)
>   rename python/qemu/{ => machine}/machine.py (98%)
>   rename python/qemu/{ => machine}/qtest.py (98%)
>   rename python/qemu/{qmp.py => qmp/__init__.py} (96%)
>   rename python/{qemu/pylintrc => setup.cfg} (50%)
>   mode change 100644 => 100755
>   create mode 100755 python/setup.py
>
Philippe Mathieu-Daudé Oct. 28, 2020, 9:37 a.m. UTC | #2
On 10/27/20 11:08 PM, John Snow wrote:
> Ping O:-)
> 
> Looking for feedback from at least Cleber and Eduardo before I barge
> ahead and send a PR to include this on master. Additional packaging and
> versioning feedback from Dan would be nice.
> 
> (I know we have a very busy two weeks here; I will continue pinging, but
> I have every intention of merging this prior to 5.2.)

Too pythonic for me, so I'll defer that to Python experts.

It would be nice to have this in 5.2 indeed...

Regards,

Phil.