mbox series

[00/29] Convert QAPI doc comments to generate rST instead of texinfo

Message ID 20200206173040.17337-1-peter.maydell@linaro.org
Headers show
Series Convert QAPI doc comments to generate rST instead of texinfo | expand

Message

Peter Maydell Feb. 6, 2020, 5:30 p.m. UTC
This series switches all our QAPI doc comments over from
texinfo format to rST.

The basic approach is somewhat similar to how we deal with kerneldoc
and hxtool: we have a custom Sphinx extension which is passed a
filename which is the json file it should run the QAPI parser over and
generate documentation for. Unlike 'kerneldoc' but somewhat like
hxtool, I have chosed to generate documentation by generating a tree
of docutils nodes, rather than by generating rST source that is then
fed to the rST parser to generate docutils nodes.  Individual lumps of
doc comment go to the rST parser, but the structured parts we render
directly. This makes it easier to get the structure and heading level
nesting correct.

Rather than trying to exactly handle all the existing comments I have
opted (as Markus suggested) to tweak them where this seemed more
sensible than contorting the rST generator to deal with
weirdnesses. The principal changes are:
 * whitespace is now significant, and multiline definitions must have
   their second and subsequent lines indented to match the first line
 * general rST format markup is permitted, not just the small set of
   markup the old texinfo generator handled. For most things (notably
   bulleted and itemized lists) the old format is the same as rST was.
 * Specific things that might trip people up:
   - instead of *bold* and _italic_ rST has **bold** and *italic*
   - lists need a preceding and following blank line
   - a lone literal '*' will need to be backslash-escaped to
     avoid a rST syntax error
 * the old leading '|' for example (literal text) blocks is replaced
   by the standard rST '::' literal block.
 * headings and subheadings must now be in a freeform documentation
   comment of their own
 * we support arbitrary levels of sub- and sub-sub-heading, not just a
   main and sub-heading like the old texinfo generator
 * as a special case, @foo is retained and is equivalent to ``foo``

This policy means that most of the patch series is cleanups to the doc
comments:
 - doc comments missing the ':' after @argument (misrenders
   in the texi, is a syntax error for rST)
 - doc comments with indent that doesn't match the tightened
   requirements (no effect on the texi, but rST cares)
 - some lists needed leading/trailing blank lines adding
 - a graph intended to be rendered as ascii-art wasn't correctly
   marked up as a literal block (misrenders in the texi, syntax error
   for rST)
 - some stray hardcoded tab characters needed changing to spaces
 - in a few places parts of a doc comment were in the wrong order
   resulting in it being rendered into the wrong section by mistake
 - a few instances of texinfo `quotes' needed to be changed to rST
   'quotes'; similarly _this_ isn't valid rST markup, and in one place
   a literal * needed backslash-escaping
 - numerous places were trying to have lists without using the list
   markup, which renders as weird run-on sentences
 - the two places which define headings in the middle of a doc comment
   are changed to put the heading definition in the
   standalone-comment-block that the new generator wants (no change to
   the texinfo output)

Patch 9 changes 663 lines in 20 files but it is purely a whitespace
change (verifiable with 'git show -w').  The other patches are
generally smaller and straightforward.

All of these patches except for the "escape a literal '*'" one at the
end have either no/minimal effect on the generated texinfo or fix
misrenderings.

Moving on to the actual code changes:
 * we start by changing the existing parser code to be more careful
   with leading whitespace: instead of stripping it all, it strips
   only the amount required for indented multiline definitions, and
   complains if it finds an unexpected de-indent. The texinfo
   generator code is updated to do whitespace stripping itself, so
   there is no change to the generated texi source.
 * then we add the new qapidoc Sphinx extension, which is not yet used
   by anything. This is a 500 line script, all added in one patch. I
   can split it if people think that would help, but I'm not sure I
   see a good split point.
 * then we can convert the two generated reference documents, one at a
   time. This is mostly just updating makefile rules and the like.
 * after that we can do some minor tweaks to doc comments that would
   have confused the texinfo parser: changing our two instances of
   '|'-markup literal blocks to rST '::' literal blocks, and adding
   some headings to the GA reference so the rST interop manual ToC
   looks better.
 * finally, we can delete the old texinfo machinery and update the
   markup docs in docs/devel/qapi-code-gen.txt

There are a few things I have left out of this initial series:
 * unlike the texinfo, there is no generation of index entries
   or an index in the HTML docs
 * although there are HTML anchors on all the command/object/etc
   headings, they are not stable but just serial-number based
   tags like '#qapidoc-35', so not suitable for trying to link
   to from other parts of the docs

My view is that we can add niceties like this later; the series
already seems big enough to me.

You can find the HTML rendered version of the results
of this series at:
http://people.linaro.org/~peter.maydell/qdoc-snapshot/interop/qemu-ga-ref.html
http://people.linaro.org/~peter.maydell/qdoc-snapshot/interop/qemu-qmp-ref.html
(look also at
 http://people.linaro.org/~peter.maydell/qdoc-snapshot/interop/index.html
 if you want to see how the ToC for the interop manual comes out)
The manpages are
http://people.linaro.org/~peter.maydell/qemu-ga-ref.7
http://people.linaro.org/~peter.maydell/qemu-qmp-ref.7
(download and render with 'man -l path/to/foo.7')

For comparison, the old texinfo-to-HTML versions of the docs are:
https://www.qemu.org/docs/master/qemu-ga-ref.html
https://www.qemu.org/docs/master/qemu-qmp-ref.html

The first four patches have already been posted separately and
reviewed.

I did at some point while working on this eyeball all the generated
documentation against the old versions, but there's an awful lot of
it, so I might have missed some minor stuff, and I didn't try to redo
that full-eyeball-diff with the absolute final version of the code. So
it's possible there are some minor misrenderings lurking, but I don't
think so.

thanks
-- PMM

Peter Maydell (29):
  configure: Allow user to specify sphinx-build binary
  configure: Check that sphinx-build is using Python 3
  Makefile: Fix typo in dependency list for interop manpages
  qga/qapi-schema.json: Fix missing '-' in GuestDiskBusType doc comment
  qga/qapi-schema.json: Fix indent level on doc comments
  qga/qapi-schema.json: minor format fixups for rST
  qapi/block-core.json: Use literal block for ascii art
  qapi: Use ':' after @argument in doc comments
  qapi: Fix indent level on doc comments in json files
  qapi: Remove hardcoded tabs
  qapi/ui.json: Put input-send-event body text in the right place
  qapi: Explicitly put "foo: dropped in n.n" notes into Notes section
  qapi/ui.json: Avoid `...' texinfo style quoting
  qapi/block-core.json: Use explicit bulleted lists
  qapi/ui.json: Use explicit bulleted lists
  qapi/{block,misc,tmp}.json: Use explicit bulleted lists
  qapi: Add blank lines before bulleted lists
  qapi/migration.json: Replace _this_ with *this*
  qapi/qapi-schema.json: Put headers in their own doc-comment blocks
  qapi/machine.json: Escape a literal '*' in doc comment
  scripts/qapi: Move doc-comment whitespace stripping to doc.py
  scripts/qapi/parser.py: improve doc comment indent handling
  docs/sphinx: Add new qapi-doc Sphinx extension
  docs/interop: Convert qemu-ga-ref to rST
  docs/interop: Convert qemu-qmp-ref to rST
  qapi: Use rST markup for literal blocks
  qga/qapi-schema.json: Add some headings
  scripts/qapi: Remove texinfo generation support
  docs/devel/qapi-code-gen.txt: Update to new rST backend conventions

 docs/devel/qapi-code-gen.txt   |   90 ++-
 configure                      |   22 +-
 Makefile                       |   58 +-
 tests/Makefile.include         |   15 +-
 qapi/block-core.json           | 1127 ++++++++++++++++----------------
 qapi/block.json                |   47 +-
 qapi/char.json                 |   10 +-
 qapi/dump.json                 |    4 +-
 qapi/introspect.json           |   12 +-
 qapi/job.json                  |   32 +-
 qapi/machine-target.json       |   18 +-
 qapi/machine.json              |   16 +-
 qapi/migration.json            |  206 +++---
 qapi/misc-target.json          |    8 +-
 qapi/misc.json                 |  138 ++--
 qapi/net.json                  |   26 +-
 qapi/qapi-schema.json          |   18 +-
 qapi/qdev.json                 |   10 +-
 qapi/qom.json                  |    4 +-
 qapi/rocker.json               |   12 +-
 qapi/run-state.json            |   34 +-
 qapi/sockets.json              |    8 +-
 qapi/tpm.json                  |    4 +-
 qapi/trace.json                |   15 +-
 qapi/transaction.json          |    4 +-
 qapi/ui.json                   |  119 ++--
 qga/qapi-schema.json           |  160 ++---
 MAINTAINERS                    |    3 +-
 docs/conf.py                   |   16 +-
 docs/index.html.in             |    2 -
 docs/interop/conf.py           |    4 +
 docs/interop/index.rst         |    2 +
 docs/interop/qemu-ga-ref.rst   |    4 +
 docs/interop/qemu-ga-ref.texi  |   80 ---
 docs/interop/qemu-qmp-ref.rst  |    4 +
 docs/interop/qemu-qmp-ref.texi |   80 ---
 docs/sphinx/qapidoc.py         |  504 ++++++++++++++
 scripts/qapi-gen.py            |    2 -
 scripts/qapi/doc.py            |  302 ---------
 scripts/qapi/gen.py            |    7 -
 scripts/qapi/parser.py         |   94 ++-
 41 files changed, 1726 insertions(+), 1595 deletions(-)
 create mode 100644 docs/interop/qemu-ga-ref.rst
 delete mode 100644 docs/interop/qemu-ga-ref.texi
 create mode 100644 docs/interop/qemu-qmp-ref.rst
 delete mode 100644 docs/interop/qemu-qmp-ref.texi
 create mode 100644 docs/sphinx/qapidoc.py
 delete mode 100644 scripts/qapi/doc.py

-- 
2.20.1

Comments

Peter Maydell Feb. 6, 2020, 6:47 p.m. UTC | #1
On Thu, 6 Feb 2020 at 17:30, Peter Maydell <peter.maydell@linaro.org> wrote:
> This series switches all our QAPI doc comments over from

> texinfo format to rST.


Oops, this fails 'make check' because I forgot to run it (not thinking
of docs changes as being stuff that affects test) and there are
a couple of golden-master files that need updating. I have some
minor changes that I'll put in v2 which correct that:

 * a new patch "tests/qapi/doc-good.json: Clean up markup"
   which removes some oddities of the old markup that we don't
   want to support, tightening up the tested doc comments:
     * in a single list the bullet types must all match
     * lists must have leading and following blank lines
     * indentation is important
     * the '|' example syntax is going to go away entirely,
       so stop testing it
 * patches "scripts/qapi: Move doc-comment whitespace stripping to doc.py"
   and "scripts/qapi/parser.py: improve doc comment indent handling"
   both make minor whitespace changes that require updates to the
   tests/qapi-schema/doc-good.out reference
 * "scripts/qapi: Remove texinfo generation support" now
   removes the no-longer-used tests/qapi-schema/doc-good.texi

I won't send v2 until this has had some review, though. If you
want a git branch with the pending-for-v2 fixups it's at:
https://git.linaro.org/people/peter.maydell/qemu-arm.git sphinx-conversions

thanks
-- PMM
no-reply@patchew.org Feb. 6, 2020, 7:53 p.m. UTC | #2
Patchew URL: https://patchew.org/QEMU/20200206173040.17337-1-peter.maydell@linaro.org/



Hi,

This series failed the docker-mingw@fedora build test. Please find the testing commands and
their output below. If you have Docker installed, you can probably reproduce it
locally.

=== TEST SCRIPT BEGIN ===
#! /bin/bash
export ARCH=x86_64
make docker-image-fedora V=1 NETWORK=1
time make docker-test-mingw@fedora J=14 NETWORK=1
=== TEST SCRIPT END ===

  GEN     docs/index.html
  CC      crypto/aes.o
No filename or title
make: *** [/tmp/qemu-test/src/rules.mak:392: docs/interop/qemu-qmp-ref.7] Error 255
make: *** Waiting for unfinished jobs....
Traceback (most recent call last):
  File "./tests/docker/docker.py", line 664, in <module>
---
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['sudo', '-n', 'docker', 'run', '--label', 'com.qemu.instance.uuid=62059c3553b64eb384aee0a720c4473b', '-u', '1003', '--security-opt', 'seccomp=unconfined', '--rm', '-e', 'TARGET_LIST=', '-e', 'EXTRA_CONFIGURE_OPTS=', '-e', 'V=', '-e', 'J=14', '-e', 'DEBUG=', '-e', 'SHOW_ENV=', '-e', 'CCACHE_DIR=/var/tmp/ccache', '-v', '/home/patchew2/.cache/qemu-docker-ccache:/var/tmp/ccache:z', '-v', '/var/tmp/patchew-tester-tmp-_3tt9kgf/src/docker-src.2020-02-06-14.51.13.6510:/var/tmp/qemu:z,ro', 'qemu:fedora', '/var/tmp/qemu/run', 'test-mingw']' returned non-zero exit status 2.
filter=--filter=label=com.qemu.instance.uuid=62059c3553b64eb384aee0a720c4473b
make[1]: *** [docker-run] Error 1
make[1]: Leaving directory `/var/tmp/patchew-tester-tmp-_3tt9kgf/src'
make: *** [docker-run-test-mingw@fedora] Error 2

real    2m2.836s
user    0m8.762s


The full log is available at
http://patchew.org/logs/20200206173040.17337-1-peter.maydell@linaro.org/testing.docker-mingw@fedora/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to patchew-devel@redhat.com
no-reply@patchew.org Feb. 6, 2020, 7:56 p.m. UTC | #3
Patchew URL: https://patchew.org/QEMU/20200206173040.17337-1-peter.maydell@linaro.org/



Hi,

This series failed the docker-quick@centos7 build test. Please find the testing commands and
their output below. If you have Docker installed, you can probably reproduce it
locally.

=== TEST SCRIPT BEGIN ===
#!/bin/bash
make docker-image-centos7 V=1 NETWORK=1
time make docker-test-quick@centos7 SHOW_ENV=1 J=14 NETWORK=1
=== TEST SCRIPT END ===

  CC      tests/test-qapi-commands.o
  CC      tests/test-string-input-visitor.o
  CC      tests/test-string-output-visitor.o
doc-good FAIL
--- /tmp/qemu-test/src/tests/qapi-schema/doc-good.out
+++ 
@@ -1,199 +0,0 @@
---
+++ 
@@ -0,0 +1 @@
+doc-good.json:104:1: unexpected de-indent
make: *** [check-tests/qapi-schema/frontend] Error 1
make: *** Waiting for unfinished jobs....
  CC      tests/test-qmp-event.o
Traceback (most recent call last):
---
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['sudo', '-n', 'docker', 'run', '--label', 'com.qemu.instance.uuid=ac8e8dbf593c42e385721554759cb252', '-u', '1003', '--security-opt', 'seccomp=unconfined', '--rm', '-e', 'TARGET_LIST=', '-e', 'EXTRA_CONFIGURE_OPTS=', '-e', 'V=', '-e', 'J=14', '-e', 'DEBUG=', '-e', 'SHOW_ENV=1', '-e', 'CCACHE_DIR=/var/tmp/ccache', '-v', '/home/patchew2/.cache/qemu-docker-ccache:/var/tmp/ccache:z', '-v', '/var/tmp/patchew-tester-tmp-y9h_kw4r/src/docker-src.2020-02-06-14.53.51.11537:/var/tmp/qemu:z,ro', 'qemu:centos7', '/var/tmp/qemu/run', 'test-quick']' returned non-zero exit status 2.
filter=--filter=label=com.qemu.instance.uuid=ac8e8dbf593c42e385721554759cb252
make[1]: *** [docker-run] Error 1
make[1]: Leaving directory `/var/tmp/patchew-tester-tmp-y9h_kw4r/src'
make: *** [docker-run-test-quick@centos7] Error 2

real    2m17.651s
user    0m7.653s


The full log is available at
http://patchew.org/logs/20200206173040.17337-1-peter.maydell@linaro.org/testing.docker-quick@centos7/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to patchew-devel@redhat.com
Markus Armbruster Feb. 7, 2020, 5 p.m. UTC | #4
Peter Maydell <peter.maydell@linaro.org> writes:

> This series switches all our QAPI doc comments over from

> texinfo format to rST.

>

> The basic approach is somewhat similar to how we deal with kerneldoc

> and hxtool: we have a custom Sphinx extension which is passed a

> filename which is the json file it should run the QAPI parser over and

> generate documentation for. Unlike 'kerneldoc' but somewhat like

> hxtool, I have chosed to generate documentation by generating a tree

> of docutils nodes, rather than by generating rST source that is then

> fed to the rST parser to generate docutils nodes.  Individual lumps of

> doc comment go to the rST parser, but the structured parts we render

> directly. This makes it easier to get the structure and heading level

> nesting correct.

>

> Rather than trying to exactly handle all the existing comments I have

> opted (as Markus suggested) to tweak them where this seemed more

> sensible than contorting the rST generator to deal with

> weirdnesses. The principal changes are:

>  * whitespace is now significant, and multiline definitions must have

>    their second and subsequent lines indented to match the first line

>  * general rST format markup is permitted, not just the small set of

>    markup the old texinfo generator handled. For most things (notably

>    bulleted and itemized lists) the old format is the same as rST was.

>  * Specific things that might trip people up:

>    - instead of *bold* and _italic_ rST has **bold** and *italic*


Actually, qapi-code-gen.txt documents and doc.py implements *strong* and
_emphasis_.  Texinfo commonly renders them as bold and italic when the
output format supports that.  rST has **strong** and *emphasis*.

Your series adjusts emphasis markup for rST [PATCH 18].  Since it
doesn't touch strong markup, strong silently becomes emphasis.  I guess
that's okay, perhaps even an improvement, but double-checking the actual
uses of this markup wouldn't hurt.

>    - lists need a preceding and following blank line

>    - a lone literal '*' will need to be backslash-escaped to

>      avoid a rST syntax error

>  * the old leading '|' for example (literal text) blocks is replaced

>    by the standard rST '::' literal block.

>  * headings and subheadings must now be in a freeform documentation

>    comment of their own


Can we simply use rST instead?  See my review of PATCH 18.

>  * we support arbitrary levels of sub- and sub-sub-heading, not just a

>    main and sub-heading like the old texinfo generator

>  * as a special case, @foo is retained and is equivalent to ``foo``


Apart from these remarks, your changes look sensible to me right now.  I
hope they'll still look that way when I'm done reviewing :)

[...]
Peter Maydell Feb. 7, 2020, 5:10 p.m. UTC | #5
On Fri, 7 Feb 2020 at 17:00, Markus Armbruster <armbru@redhat.com> wrote:
>

> Peter Maydell <peter.maydell@linaro.org> writes:

>

> > This series switches all our QAPI doc comments over from

> > texinfo format to rST.

> >

> > The basic approach is somewhat similar to how we deal with kerneldoc

> > and hxtool: we have a custom Sphinx extension which is passed a

> > filename which is the json file it should run the QAPI parser over and

> > generate documentation for. Unlike 'kerneldoc' but somewhat like

> > hxtool, I have chosed to generate documentation by generating a tree

> > of docutils nodes, rather than by generating rST source that is then

> > fed to the rST parser to generate docutils nodes.  Individual lumps of

> > doc comment go to the rST parser, but the structured parts we render

> > directly. This makes it easier to get the structure and heading level

> > nesting correct.

> >

> > Rather than trying to exactly handle all the existing comments I have

> > opted (as Markus suggested) to tweak them where this seemed more

> > sensible than contorting the rST generator to deal with

> > weirdnesses. The principal changes are:

> >  * whitespace is now significant, and multiline definitions must have

> >    their second and subsequent lines indented to match the first line

> >  * general rST format markup is permitted, not just the small set of

> >    markup the old texinfo generator handled. For most things (notably

> >    bulleted and itemized lists) the old format is the same as rST was.

> >  * Specific things that might trip people up:

> >    - instead of *bold* and _italic_ rST has **bold** and *italic*

>

> Actually, qapi-code-gen.txt documents and doc.py implements *strong* and

> _emphasis_.  Texinfo commonly renders them as bold and italic when the

> output format supports that.  rST has **strong** and *emphasis*.

>

> Your series adjusts emphasis markup for rST [PATCH 18].  Since it

> doesn't touch strong markup, strong silently becomes emphasis.  I guess

> that's okay, perhaps even an improvement, but double-checking the actual

> uses of this markup wouldn't hurt.


Yeah, that would be a good plan.
 git grep '\*[^*]*\*' qapi/*.json
(and eyeball-filtering out the false hits) shows just one use:

qapi/introspect.json:# Note: the QAPI schema is also used to help
define *internal*

I can put a patch on the end which converts that to **internal**
once the rST generator is in use.

> >    - lists need a preceding and following blank line

> >    - a lone literal '*' will need to be backslash-escaped to

> >      avoid a rST syntax error

> >  * the old leading '|' for example (literal text) blocks is replaced

> >    by the standard rST '::' literal block.

> >  * headings and subheadings must now be in a freeform documentation

> >    comment of their own

>

> Can we simply use rST instead?  See my review of PATCH 18.


No, we can't (see my reply to that review). In theory you could have
the heading syntax be a rST heading, but that is not feasible to
recognise in the Python script[*] and it gives the impression that
it is just an inline rST heading, not something that's more complicated
and structured.

[*] You'd need to manually re-implement the weird thing rST does
where practically any kind of underlining is valid and it figures
out which one means which depth by looking at the usage through
the whole document. You'd have to do bizarre stuff like running
through the entire set of doc comments once doing no output but
just looking at heading underline characters to guess which one
is which depth, and then once you'd figured that out you could
do it all over again actually generating the doc.

thanks
-- PMM
Markus Armbruster Feb. 8, 2020, 2:15 p.m. UTC | #6
Peter Maydell <peter.maydell@linaro.org> writes:

> On Fri, 7 Feb 2020 at 17:00, Markus Armbruster <armbru@redhat.com> wrote:

>>

>> Peter Maydell <peter.maydell@linaro.org> writes:

>>

>> > This series switches all our QAPI doc comments over from

>> > texinfo format to rST.

>> >

>> > The basic approach is somewhat similar to how we deal with kerneldoc

>> > and hxtool: we have a custom Sphinx extension which is passed a

>> > filename which is the json file it should run the QAPI parser over and

>> > generate documentation for. Unlike 'kerneldoc' but somewhat like

>> > hxtool, I have chosed to generate documentation by generating a tree

>> > of docutils nodes, rather than by generating rST source that is then

>> > fed to the rST parser to generate docutils nodes.  Individual lumps of

>> > doc comment go to the rST parser, but the structured parts we render

>> > directly. This makes it easier to get the structure and heading level

>> > nesting correct.

>> >

>> > Rather than trying to exactly handle all the existing comments I have

>> > opted (as Markus suggested) to tweak them where this seemed more

>> > sensible than contorting the rST generator to deal with

>> > weirdnesses. The principal changes are:

>> >  * whitespace is now significant, and multiline definitions must have

>> >    their second and subsequent lines indented to match the first line

>> >  * general rST format markup is permitted, not just the small set of

>> >    markup the old texinfo generator handled. For most things (notably

>> >    bulleted and itemized lists) the old format is the same as rST was.

>> >  * Specific things that might trip people up:

>> >    - instead of *bold* and _italic_ rST has **bold** and *italic*

>>

>> Actually, qapi-code-gen.txt documents and doc.py implements *strong* and

>> _emphasis_.  Texinfo commonly renders them as bold and italic when the

>> output format supports that.  rST has **strong** and *emphasis*.

>>

>> Your series adjusts emphasis markup for rST [PATCH 18].  Since it

>> doesn't touch strong markup, strong silently becomes emphasis.  I guess

>> that's okay, perhaps even an improvement, but double-checking the actual

>> uses of this markup wouldn't hurt.

>

> Yeah, that would be a good plan.

>  git grep '\*[^*]*\*' qapi/*.json

> (and eyeball-filtering out the false hits) shows just one use:

>

> qapi/introspect.json:# Note: the QAPI schema is also used to help

> define *internal*

>

> I can put a patch on the end which converts that to **internal**

> once the rST generator is in use.


I wrote that one, and I think changing it from strong to emphasis is an
improvement.  Let's point to it in the commit message, and call it a
day.

>> >    - lists need a preceding and following blank line

>> >    - a lone literal '*' will need to be backslash-escaped to

>> >      avoid a rST syntax error

>> >  * the old leading '|' for example (literal text) blocks is replaced

>> >    by the standard rST '::' literal block.

>> >  * headings and subheadings must now be in a freeform documentation

>> >    comment of their own

>>

>> Can we simply use rST instead?  See my review of PATCH 18.

>

> No, we can't (see my reply to that review). In theory you could have

> the heading syntax be a rST heading, but that is not feasible to

> recognise in the Python script[*] and it gives the impression that

> it is just an inline rST heading, not something that's more complicated

> and structured.

>

> [*] You'd need to manually re-implement the weird thing rST does

> where practically any kind of underlining is valid and it figures

> out which one means which depth by looking at the usage through

> the whole document. You'd have to do bizarre stuff like running

> through the entire set of doc comments once doing no output but

> just looking at heading underline characters to guess which one

> is which depth, and then once you'd figured that out you could

> do it all over again actually generating the doc.


I understand the difficulty of parsing rST (Paolo called it "the Perl of
markup languages" for a reason).  What I don't yet understand is (1) why
we need to recognize the document structure of doc comments, and (2) why
we can do that by recognizing '=' markup, but ignore the native rST
document structure markup.
Peter Maydell Feb. 8, 2020, 2:59 p.m. UTC | #7
On Sat, 8 Feb 2020 at 14:16, Markus Armbruster <armbru@redhat.com> wrote:
> I understand the difficulty of parsing rST (Paolo called it "the Perl of

> markup languages" for a reason).  What I don't yet understand is (1) why

> we need to recognize the document structure of doc comments, and (2) why

> we can do that by recognizing '=' markup, but ignore the native rST

> document structure markup.


I think we're completely at cross purposes here, so there's
something I've not managed to communicate clearly.

We don't need to recognize the document structure of doc
comments, indeed my implementation does not -- it just
throws a doc comment at the rST parser to handle.

We *do* need to recognize the structure *of the document*
(ie that it does have a thing with a heading "Block devices"
that contains another thing with a heading "Background jobs"
that in turn contains documentation for JobType, JobStatus....)
so that when we're building up our tree of docutils node
objects we know when we need to create a new 'section'
node and give it a title 'Block devices' and which of
the various section nodes in the tree should have all
the nodes that make up the documentation of 'JobType'
added to it.

In order to achieve this separation (don't care about
document structure inside lumps of rST, but do want to
know what the overall section/subsection structure of
the document is), this patchset pulls the identification
of the document structure (heading/subheadings) completely
out of being something you might find in the middle of
a doc comment, and makes them their own
special kind of markup:

##
# = This is a heading
##

(In my head I find I'm thinking of this as "not actually
a doc comment", which is probably where some of my
lack of clarity is coming from, since syntactically
speaking and from the point of view of qapi/parser.py
that is a sort of doc comment.)

I suppose that there's an argument that the identification
of headings and subheadings should really be done in
qapi/parser.py, so that instead of

        vis = QAPISchemaGenRSTVisitor(self)
        vis.visit_begin(schema)
        for doc in schema.docs:
            if doc.symbol:
                vis.symbol(doc, schema.lookup_entity(doc.symbol))
            else:
                vis.freeform(doc)

you would have something more like

        vis = QAPISchemaGenRSTVisitor(self)
        vis.visit_begin(schema)
        for doc in schema.docs:
            if doc.symbol:
                vis.symbol(doc, schema.lookup_entity(doc.symbol))
            else if doc.is_section_header:
                vis.start_new_section(doc)
            else:
                vis.freeform(doc)

(with the identification of headers and pulling out of
what level of nesting this header is and what its text
is done in parser.py)

thanks
-- PMM
Aleksandar Markovic Feb. 10, 2020, 12:34 a.m. UTC | #8
On Thursday, February 6, 2020, Peter Maydell <peter.maydell@linaro.org>
wrote:

>

> > This series switches all our QAPI doc > > > comments over from

> > texinfo format to rST.

>

>

Regardeless of the outcome of the discussions over this series, I just want
to say that I support it as a potential user of the document (there is no
"Supported-by:" mark or similar, so this is all I can do). Also, as a
general participant in QEMU community,  I seems to me this is a significant
step in shaping up QEMU documentation.

Best regards,
Aleksandar



>

>
<br><br>On Thursday, February 6, 2020, Peter Maydell &lt;<a href="mailto:peter.maydell@linaro.org">peter.maydell@linaro.org</a>&gt; wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br>&gt; This series switches all our QAPI doc &gt; &gt; &gt; comments over from<br>&gt; texinfo format to rST.<br>
<br></blockquote><div><br></div><div>Regardeless of the outcome of the discussions over this series, I just want to say that I support it as a potential user of the document (there is no &quot;Supported-by:&quot; mark or similar, so this is all I can do). Also, as a general participant in QEMU community,  I seems to me this is a significant step in shaping up QEMU documentation.</div><div><br></div><div>Best regards,</div><div>Aleksandar</div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
</blockquote>