[RFC,00/10] Add kernel-doc support to our Sphinx setup

Hi; this is an RFC patchset which I'm sending out mostly to provide a
work-in-progress snapshot that might be useful for Gabriel's GSoC
project on API documentation generation. You can find the equivalent
git branch here:
https://git.linaro.org/people/peter.maydell/qemu-arm.git/log/?h=sphinx-docs
(git url/branch for cloning:
  https://git.linaro.org/people/peter.maydell/qemu-arm.git sphinx-docs )

The patchset starts by pulling in the kernel-doc script from the
kernel sources and wiring it up in our Sphinx config.  The version of
the script is just from head-of-kernel-git from when I was working on
this at the beginning of the year; we should probably check whether
we should update it to something newer and/or something from a more
specific kernel release.

The patchset then demonstrates the functionality with two documents:
a standalone file that just has API docs of the functions from
bitops.h, and an extension of the existing memory.rst doc to include
a "detailed API docs" section at the end with the generated docs from
memory.h.  In both cases to get the build to not error out we need to
fix up syntax issues in the header comments.  For the memory.h stuff
in particular some of the fixes are a bit ropy, and I also ended up
using a big "disable nitpick mode" hammer to shut up an error
(leaving some non-fatal warnings still generated).

Broadly speaking the earlier patches in the series are more
"solid" than the later ones which are a little more hacky.

Some specific notes:

(1) our Makefile rune runs sphinx-build with -n, which enables
"nitpick mode", where sphinx will complain about references that it
wanted to hyperlink but which it doesn't have anywhere te link them
to.  This is something that I think I got from the kernel's makefile
rules, and it's obviously nice for catching typos, but sphinx also
complains about references to "uint8_t" and other system types
because obviously there are no definitions of those in our headers. 
I have a patch in there which tries to whitelist these, but this may
not be the best approach. (Nitpick mode might also be awkward if we
want to generate docs for API A which uses some types from API B
which we have not yet started generating docs for.)

(2) kernel-doc assumes the kernel C coding style which always uses
explicit 'struct'.  This clashes with QEMU's style which usually
hides 'struct' behind typedefs, and seems to result in needing to
write 'struct' in the doc comment to get it recognised as being
documentation of the struct (see eg "memory.h: attempted kernel-doc
fixes") and also failure to recognize that "#MemoryRegion" is a
reference to the MemoryRegion struct type (resulting in errors about
references not being found).  I'm not sure exactly what to do here --
it may be some combination of making kernel-doc more flexible in
coping with structs that hide behind typedefs, and fixing up syntax
in our doc comments.

(3) There are definitely some other syntax issues that still need
fixing -- for instance if you look at the generated documentation for
memory_region_init_ram_nomigrate() the "Description" section is just
a sentence fragment, because the doc comment has been misparsed due
to the way the descriptive text following "function() - " is split
over multiple lines.  We would need to do a more thorough
proofreading check and fix up the syntax as needed.

(4) Some Sphinx warnings generated are legitimate -- it complains
correctly that we don't have doc comments covering all the fields in
some of the structs in memory.h, for instance.  In an ideal world we
would fix these too :-)

thanks
-- PMM

Peter Maydell (10):
  scripts/kernel-doc: Import kernel-doc script from Linux
  docs: Add kerneldoc sphinx module from Linux
  docs/conf.py: Enable use of kerneldoc sphinx extension
  docs/conf.py: Ignore some missing references in nitpick mode
  bitops.h: Silence kernel-doc complaints
  docs: Create bitops.rst as example of kernel-docs
  memory.h: Fix kerneldoc syntax issues
  docs: Add kernel-doc API documentation to memory.rst
  memory.h: attempted kernel-doc fixes
  Makefile: disable Sphinx nitpicking

 Makefile                 |    2 +-
 include/exec/memory.h    |   31 +-
 include/qemu/bitops.h    |   52 +-
 docs/conf.py             |   22 +-
 docs/devel/bitops.rst    |    8 +
 docs/devel/index.rst     |    1 +
 docs/devel/memory.rst    |    5 +
 docs/sphinx/kerneldoc.py |  150 +++
 scripts/kernel-doc       | 2223 ++++++++++++++++++++++++++++++++++++++
 9 files changed, 2458 insertions(+), 36 deletions(-)
 create mode 100644 docs/devel/bitops.rst
 create mode 100644 docs/sphinx/kerneldoc.py
 create mode 100755 scripts/kernel-doc

-- 
2.20.1

Patchew URL: https://patchew.org/QEMU/20190521122519.12573-1-peter.maydell@linaro.org/



Hi,

This series seems to have some coding style problems. See output below for
more information:

Subject: [Qemu-devel] [RFC 00/10] Add kernel-doc support to our Sphinx setup
Message-id: 20190521122519.12573-1-peter.maydell@linaro.org
Type: series

=== TEST SCRIPT BEGIN ===
#!/bin/bash
git rev-parse base > /dev/null || exit 0
git config --local diff.renamelimit 0
git config --local diff.renames True
git config --local diff.algorithm histogram
./scripts/checkpatch.pl --mailback base..
=== TEST SCRIPT END ===

From https://github.com/patchew-project/qemu
 * [new tag]               patchew/20190521122519.12573-1-peter.maydell@linaro.org -> patchew/20190521122519.12573-1-peter.maydell@linaro.org
Switched to a new branch 'test'
02a0d0d337 Makefile: disable Sphinx nitpicking
2124e52316 memory.h: attempted kernel-doc fixes
f73463aa82 docs: Add kernel-doc API documentation to memory.rst
e775bbb9db memory.h: Fix kerneldoc syntax issues
b7a8bdc1ba docs: Create bitops.rst as example of kernel-docs
19ff73c38a bitops.h: Silence kernel-doc complaints
5a4c54aa09 docs/conf.py: Ignore some missing references in nitpick mode
a08b2a1c93 docs/conf.py: Enable use of kerneldoc sphinx extension
c4f9da74ce docs: Add kerneldoc sphinx module from Linux
e97b2d12d4 scripts/kernel-doc: Import kernel-doc script from Linux

=== OUTPUT BEGIN ===
1/10 Checking commit e97b2d12d441 (scripts/kernel-doc: Import kernel-doc script from Linux)
WARNING: added, moved or deleted file(s), does MAINTAINERS need updating?
#13: 
new file mode 100755

total: 0 errors, 1 warnings, 2223 lines checked

Patch 1/10 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
2/10 Checking commit c4f9da74cecb (docs: Add kerneldoc sphinx module from Linux)
WARNING: added, moved or deleted file(s), does MAINTAINERS need updating?
#14: 
new file mode 100644

WARNING: line over 80 characters
#84: FILE: docs/sphinx/kerneldoc.py:66:
+        tab_width = self.options.get('tab-width', self.state.document.settings.tab_width)

WARNING: line over 80 characters
#113: FILE: docs/sphinx/kerneldoc.py:95:
+            p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)

ERROR: line over 90 characters
#121: FILE: docs/sphinx/kerneldoc.py:103:
+                env.app.warn('kernel-doc \'%s\' failed with return code %d' % (" ".join(cmd), p.returncode))

WARNING: line over 80 characters
#122: FILE: docs/sphinx/kerneldoc.py:104:
+                return [nodes.error(None, nodes.paragraph(text = "kernel-doc missing"))]

WARNING: line over 80 characters
#126: FILE: docs/sphinx/kerneldoc.py:108:
+            lines = statemachine.string2lines(out, tab_width, convert_whitespace=True)

ERROR: line over 90 characters
#142: FILE: docs/sphinx/kerneldoc.py:124:
+            buf = self.state.memo.title_styles, self.state.memo.section_level, self.state.memo.reporter

WARNING: line over 80 characters
#143: FILE: docs/sphinx/kerneldoc.py:125:
+            self.state.memo.reporter = AutodocReporter(result, self.state.memo.reporter)

ERROR: line over 90 characters
#148: FILE: docs/sphinx/kerneldoc.py:130:
+                self.state.memo.title_styles, self.state.memo.section_level, self.state.memo.reporter = buf

WARNING: line over 80 characters
#155: FILE: docs/sphinx/kerneldoc.py:137:
+            return [nodes.error(None, nodes.paragraph(text = "kernel-doc missing"))]

total: 3 errors, 7 warnings, 150 lines checked

Patch 2/10 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.

3/10 Checking commit a08b2a1c931a (docs/conf.py: Enable use of kerneldoc sphinx extension)
4/10 Checking commit 5a4c54aa09f6 (docs/conf.py: Ignore some missing references in nitpick mode)
5/10 Checking commit 19ff73c38a62 (bitops.h: Silence kernel-doc complaints)
6/10 Checking commit b7a8bdc1ba62 (docs: Create bitops.rst as example of kernel-docs)
WARNING: added, moved or deleted file(s), does MAINTAINERS need updating?
#16: 
new file mode 100644

total: 0 errors, 1 warnings, 12 lines checked

Patch 6/10 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
7/10 Checking commit e775bbb9db99 (memory.h: Fix kerneldoc syntax issues)
8/10 Checking commit f73463aa82c3 (docs: Add kernel-doc API documentation to memory.rst)
9/10 Checking commit 2124e5231685 (memory.h: attempted kernel-doc fixes)
WARNING: line over 80 characters
#35: FILE: include/exec/memory.h:228:
+ * -  @attrs_to_index returns the IOMMU index for a set of transaction attributes

WARNING: line over 80 characters
#46: FILE: include/exec/memory.h:403:
+ * struct MemoryListener: callbacks structure for updates to the physical memory map

WARNING: line over 80 characters
#55: FILE: include/exec/memory.h:437:
+ * struct AddressSpace: describes a mapping of addresses to #MemoryRegion objects

ERROR: Missing Signed-off-by: line(s)

total: 1 errors, 3 warnings, 64 lines checked

Patch 9/10 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.

10/10 Checking commit 02a0d0d33747 (Makefile: disable Sphinx nitpicking)
=== OUTPUT END ===

Test command exited with code: 1


The full log is available at
http://patchew.org/logs/20190521122519.12573-1-peter.maydell@linaro.org/testing.checkpatch/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to patchew-devel@redhat.com

On Tue, 21 May 2019 at 13:25, Peter Maydell <peter.maydell@linaro.org> wrote:
>

> Hi; this is an RFC patchset which I'm sending out mostly to provide a

> work-in-progress snapshot


...patchew tells me it was sufficiently work-in-progress
that I forgot some signed-off-by lines, so for the
avoidance of doubt, all patches in this series can
have my
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>


thanks
-- PMM