[00/30] kconfig: move compiler capability tests to Kconfig

Message ID 1523595999-27433-1-git-send-email-yamada.masahiro@socionext.com
Headers show
Series
  • kconfig: move compiler capability tests to Kconfig
Related show

Message

Masahiro Yamada April 13, 2018, 5:06 a.m.
[Introduction]

The motivation of this work is to move the compiler option tests to
Kconfig from Makefile.  A number of kernel features require the
compiler support.  Enabling such features blindly in Kconfig ends up
with a lot of nasty build-time testing in Makefiles.  If a chosen
feature turns out unsupported by the compiler, what the build system
can do is either to disable it (silently!) or to forcibly break the
build, despite Kconfig has let the user to enable it.

[Major Changes in V3]

This version looks more like Make.

  - Use = operator instead of 'macro' keyword
    to define a user-defined function.

  - 'Recursively expanded variable' is implemented as a side-effect.
     A variable is a function with zero argument.

  - Support simply expanded variable which is defined by := operator

  - Support += operator.
    Probably, this feature will be useful to accumulate compiler flags.
    At least, Clang needs some prerequisite flags such as triplet
    to test other compiler flags.

  - Support $(info ...) and $(warning ...) built-in functions,
    which were useful while I was debugging this.

  - Add documentation

  - Add unit tests

  - Collect helpers to scripts/Kconfig.include

[TODO items]

I have not been able to finish this.  Maybe aiming for v4.18-rc1.

  - The text expansion should be able to defer argument expansion.
    This is necessary if we want to implement $(if ...) built-in function.

  - Error out recursive expansion of itself.
    For example,
        X = $(X)
    refers itself.
    If referenced, the expansion continues eternally.  This must be terminated.

  - More compatible behavior?
    Kconfig works mostly like Make, but there are naive cases
    where the behavior is different.

[Old Versions]

V2:  https://lkml.org/lkml/2018/2/16/610
V1:  https://lkml.org/lkml/2018/2/16/610
RFC: https://lkml.org/lkml/2018/2/8/429



Masahiro Yamada (30):
  gcc-plugins: fix build condition of SANCOV plugin
  kbuild: remove kbuild cache
  kbuild: remove CONFIG_CROSS_COMPILE support
  kconfig: reference environment variables directly and remove 'option
    env='
  kconfig: remove string expansion in file_lookup()
  kconfig: remove string expansion for mainmenu after yyparse()
  kconfig: remove sym_expand_string_value()
  kconfig: add built-in function support
  kconfig: add 'shell' built-in function
  kconfig: replace $(UNAME_RELEASE) with function call
  kconfig: begin PARAM state only when seeing a command keyword
  kconfig: support variable and user-defined function
  kconfig: support simply expanded variable
  kconfig: support append assignment operator
  kconfig: expand lefthand side of assignment statement
  kconfig: add 'info' and 'warning' built-in functions
  Documentation: kconfig: document a new Kconfig macro language
  kconfig: test: test text expansion
  kconfig: show compiler version text in the top comment
  kconfig: add basic helper macros to scripts/Kconfig.include
  stack-protector: test compiler capability in Kconfig and drop AUTO
    mode
  kconfig: add CC_IS_GCC and GCC_VERSION
  kconfig: add CC_IS_CLANG and CLANG_VERSION
  gcov: remove CONFIG_GCOV_FORMAT_AUTODETECT
  kcov: test compiler capability in Kconfig and correct dependency
  gcc-plugins: move GCC version check for PowerPC to Kconfig
  gcc-plugins: test plugin support in Kconfig and clean up Makefile
  gcc-plugins: allow to enable GCC_PLUGINS for COMPILE_TEST
  arm64: move GCC version check for ARCH_SUPPORTS_INT128 to Kconfig
  kbuild: test dead code/data elimination support in Kconfig

 Documentation/kbuild/kconfig-language.txt          |   8 -
 Documentation/kbuild/kconfig-macro-language.txt    | 179 +++++++
 Kconfig                                            |  10 +-
 MAINTAINERS                                        |   3 +-
 Makefile                                           | 113 +---
 arch/Kconfig                                       |  53 +-
 arch/arm64/Kconfig                                 |   1 +
 arch/arm64/Makefile                                |   2 -
 arch/powerpc/Kconfig                               |   2 +-
 arch/sh/Kconfig                                    |   4 +-
 arch/sparc/Kconfig                                 |   4 +-
 arch/um/Kconfig.common                             |   4 -
 arch/x86/Kconfig                                   |  15 +-
 arch/x86/um/Kconfig                                |   6 +-
 init/Kconfig                                       |  40 +-
 kernel/gcov/Kconfig                                |  17 +-
 kernel/gcov/Makefile                               |   2 -
 lib/Kconfig.debug                                  |  11 +-
 scripts/Kbuild.include                             | 101 +---
 scripts/Kconfig.include                            |  20 +
 scripts/Makefile.gcc-plugins                       |  91 +---
 scripts/Makefile.kcov                              |  10 +-
 scripts/clang-version.sh                           |  18 +-
 scripts/gcc-plugins/Makefile                       |   1 +
 scripts/gcc-x86_32-has-stack-protector.sh          |   7 +-
 scripts/gcc-x86_64-has-stack-protector.sh          |   5 -
 scripts/kconfig/confdata.c                         |  31 +-
 scripts/kconfig/kconf_id.c                         |   1 -
 scripts/kconfig/lkc.h                              |   4 -
 scripts/kconfig/lkc_proto.h                        |  15 +-
 scripts/kconfig/menu.c                             |   3 -
 scripts/kconfig/preprocess.c                       | 594 +++++++++++++++++++++
 scripts/kconfig/symbol.c                           | 109 ----
 .../kconfig/tests/preprocess/builtin_func/Kconfig  |  19 +
 .../tests/preprocess/builtin_func/__init__.py      |   8 +
 .../tests/preprocess/builtin_func/expected_stderr  |   4 +
 .../tests/preprocess/builtin_func/expected_stdout  |   1 +
 scripts/kconfig/tests/preprocess/escape/Kconfig    |  49 ++
 .../kconfig/tests/preprocess/escape/__init__.py    |   9 +
 .../tests/preprocess/escape/expected_stderr        |  10 +
 scripts/kconfig/tests/preprocess/user_func/Kconfig |  19 +
 .../kconfig/tests/preprocess/user_func/__init__.py |   7 +
 .../tests/preprocess/user_func/expected_stderr     |   5 +
 scripts/kconfig/tests/preprocess/variable/Kconfig  |  39 ++
 .../kconfig/tests/preprocess/variable/__init__.py  |   7 +
 .../tests/preprocess/variable/expected_stderr      |   7 +
 scripts/kconfig/util.c                             |  22 +-
 scripts/kconfig/zconf.l                            |  95 +++-
 scripts/kconfig/zconf.y                            |  48 +-
 49 files changed, 1259 insertions(+), 574 deletions(-)
 create mode 100644 Documentation/kbuild/kconfig-macro-language.txt
 create mode 100644 scripts/Kconfig.include
 create mode 100644 scripts/kconfig/preprocess.c
 create mode 100644 scripts/kconfig/tests/preprocess/builtin_func/Kconfig
 create mode 100644 scripts/kconfig/tests/preprocess/builtin_func/__init__.py
 create mode 100644 scripts/kconfig/tests/preprocess/builtin_func/expected_stderr
 create mode 100644 scripts/kconfig/tests/preprocess/builtin_func/expected_stdout
 create mode 100644 scripts/kconfig/tests/preprocess/escape/Kconfig
 create mode 100644 scripts/kconfig/tests/preprocess/escape/__init__.py
 create mode 100644 scripts/kconfig/tests/preprocess/escape/expected_stderr
 create mode 100644 scripts/kconfig/tests/preprocess/user_func/Kconfig
 create mode 100644 scripts/kconfig/tests/preprocess/user_func/__init__.py
 create mode 100644 scripts/kconfig/tests/preprocess/user_func/expected_stderr
 create mode 100644 scripts/kconfig/tests/preprocess/variable/Kconfig
 create mode 100644 scripts/kconfig/tests/preprocess/variable/__init__.py
 create mode 100644 scripts/kconfig/tests/preprocess/variable/expected_stderr

-- 
2.7.4

Comments

Masahiro Yamada April 13, 2018, 5:17 a.m. | #1
2018-04-13 14:06 GMT+09:00 Masahiro Yamada <yamada.masahiro@socionext.com>:



I forgot to prefix the patch subjects with the version.

This series is V3.







> [Introduction]

>

> The motivation of this work is to move the compiler option tests to

> Kconfig from Makefile.  A number of kernel features require the

> compiler support.  Enabling such features blindly in Kconfig ends up

> with a lot of nasty build-time testing in Makefiles.  If a chosen

> feature turns out unsupported by the compiler, what the build system

> can do is either to disable it (silently!) or to forcibly break the

> build, despite Kconfig has let the user to enable it.

>

> [Major Changes in V3]

>

> This version looks more like Make.

>

>   - Use = operator instead of 'macro' keyword

>     to define a user-defined function.

>

>   - 'Recursively expanded variable' is implemented as a side-effect.

>      A variable is a function with zero argument.

>

>   - Support simply expanded variable which is defined by := operator

>

>   - Support += operator.

>     Probably, this feature will be useful to accumulate compiler flags.

>     At least, Clang needs some prerequisite flags such as triplet

>     to test other compiler flags.

>

>   - Support $(info ...) and $(warning ...) built-in functions,

>     which were useful while I was debugging this.

>

>   - Add documentation

>

>   - Add unit tests

>

>   - Collect helpers to scripts/Kconfig.include

>

> [TODO items]

>

> I have not been able to finish this.  Maybe aiming for v4.18-rc1.

>

>   - The text expansion should be able to defer argument expansion.

>     This is necessary if we want to implement $(if ...) built-in function.

>

>   - Error out recursive expansion of itself.

>     For example,

>         X = $(X)

>     refers itself.

>     If referenced, the expansion continues eternally.  This must be terminated.

>

>   - More compatible behavior?

>     Kconfig works mostly like Make, but there are naive cases

>     where the behavior is different.

>

> [Old Versions]

>

> V2:  https://lkml.org/lkml/2018/2/16/610

> V1:  https://lkml.org/lkml/2018/2/16/610

> RFC: https://lkml.org/lkml/2018/2/8/429

>

>

>

> Masahiro Yamada (30):

>   gcc-plugins: fix build condition of SANCOV plugin

>   kbuild: remove kbuild cache

>   kbuild: remove CONFIG_CROSS_COMPILE support

>   kconfig: reference environment variables directly and remove 'option

>     env='

>   kconfig: remove string expansion in file_lookup()

>   kconfig: remove string expansion for mainmenu after yyparse()

>   kconfig: remove sym_expand_string_value()

>   kconfig: add built-in function support

>   kconfig: add 'shell' built-in function

>   kconfig: replace $(UNAME_RELEASE) with function call

>   kconfig: begin PARAM state only when seeing a command keyword

>   kconfig: support variable and user-defined function

>   kconfig: support simply expanded variable

>   kconfig: support append assignment operator

>   kconfig: expand lefthand side of assignment statement

>   kconfig: add 'info' and 'warning' built-in functions

>   Documentation: kconfig: document a new Kconfig macro language

>   kconfig: test: test text expansion

>   kconfig: show compiler version text in the top comment

>   kconfig: add basic helper macros to scripts/Kconfig.include

>   stack-protector: test compiler capability in Kconfig and drop AUTO

>     mode

>   kconfig: add CC_IS_GCC and GCC_VERSION

>   kconfig: add CC_IS_CLANG and CLANG_VERSION

>   gcov: remove CONFIG_GCOV_FORMAT_AUTODETECT

>   kcov: test compiler capability in Kconfig and correct dependency

>   gcc-plugins: move GCC version check for PowerPC to Kconfig

>   gcc-plugins: test plugin support in Kconfig and clean up Makefile

>   gcc-plugins: allow to enable GCC_PLUGINS for COMPILE_TEST

>   arm64: move GCC version check for ARCH_SUPPORTS_INT128 to Kconfig

>   kbuild: test dead code/data elimination support in Kconfig

>

>  Documentation/kbuild/kconfig-language.txt          |   8 -

>  Documentation/kbuild/kconfig-macro-language.txt    | 179 +++++++

>  Kconfig                                            |  10 +-

>  MAINTAINERS                                        |   3 +-

>  Makefile                                           | 113 +---

>  arch/Kconfig                                       |  53 +-

>  arch/arm64/Kconfig                                 |   1 +

>  arch/arm64/Makefile                                |   2 -

>  arch/powerpc/Kconfig                               |   2 +-

>  arch/sh/Kconfig                                    |   4 +-

>  arch/sparc/Kconfig                                 |   4 +-

>  arch/um/Kconfig.common                             |   4 -

>  arch/x86/Kconfig                                   |  15 +-

>  arch/x86/um/Kconfig                                |   6 +-

>  init/Kconfig                                       |  40 +-

>  kernel/gcov/Kconfig                                |  17 +-

>  kernel/gcov/Makefile                               |   2 -

>  lib/Kconfig.debug                                  |  11 +-

>  scripts/Kbuild.include                             | 101 +---

>  scripts/Kconfig.include                            |  20 +

>  scripts/Makefile.gcc-plugins                       |  91 +---

>  scripts/Makefile.kcov                              |  10 +-

>  scripts/clang-version.sh                           |  18 +-

>  scripts/gcc-plugins/Makefile                       |   1 +

>  scripts/gcc-x86_32-has-stack-protector.sh          |   7 +-

>  scripts/gcc-x86_64-has-stack-protector.sh          |   5 -

>  scripts/kconfig/confdata.c                         |  31 +-

>  scripts/kconfig/kconf_id.c                         |   1 -

>  scripts/kconfig/lkc.h                              |   4 -

>  scripts/kconfig/lkc_proto.h                        |  15 +-

>  scripts/kconfig/menu.c                             |   3 -

>  scripts/kconfig/preprocess.c                       | 594 +++++++++++++++++++++

>  scripts/kconfig/symbol.c                           | 109 ----

>  .../kconfig/tests/preprocess/builtin_func/Kconfig  |  19 +

>  .../tests/preprocess/builtin_func/__init__.py      |   8 +

>  .../tests/preprocess/builtin_func/expected_stderr  |   4 +

>  .../tests/preprocess/builtin_func/expected_stdout  |   1 +

>  scripts/kconfig/tests/preprocess/escape/Kconfig    |  49 ++

>  .../kconfig/tests/preprocess/escape/__init__.py    |   9 +

>  .../tests/preprocess/escape/expected_stderr        |  10 +

>  scripts/kconfig/tests/preprocess/user_func/Kconfig |  19 +

>  .../kconfig/tests/preprocess/user_func/__init__.py |   7 +

>  .../tests/preprocess/user_func/expected_stderr     |   5 +

>  scripts/kconfig/tests/preprocess/variable/Kconfig  |  39 ++

>  .../kconfig/tests/preprocess/variable/__init__.py  |   7 +

>  .../tests/preprocess/variable/expected_stderr      |   7 +

>  scripts/kconfig/util.c                             |  22 +-

>  scripts/kconfig/zconf.l                            |  95 +++-

>  scripts/kconfig/zconf.y                            |  48 +-

>  49 files changed, 1259 insertions(+), 574 deletions(-)

>  create mode 100644 Documentation/kbuild/kconfig-macro-language.txt

>  create mode 100644 scripts/Kconfig.include

>  create mode 100644 scripts/kconfig/preprocess.c

>  create mode 100644 scripts/kconfig/tests/preprocess/builtin_func/Kconfig

>  create mode 100644 scripts/kconfig/tests/preprocess/builtin_func/__init__.py

>  create mode 100644 scripts/kconfig/tests/preprocess/builtin_func/expected_stderr

>  create mode 100644 scripts/kconfig/tests/preprocess/builtin_func/expected_stdout

>  create mode 100644 scripts/kconfig/tests/preprocess/escape/Kconfig

>  create mode 100644 scripts/kconfig/tests/preprocess/escape/__init__.py

>  create mode 100644 scripts/kconfig/tests/preprocess/escape/expected_stderr

>  create mode 100644 scripts/kconfig/tests/preprocess/user_func/Kconfig

>  create mode 100644 scripts/kconfig/tests/preprocess/user_func/__init__.py

>  create mode 100644 scripts/kconfig/tests/preprocess/user_func/expected_stderr

>  create mode 100644 scripts/kconfig/tests/preprocess/variable/Kconfig

>  create mode 100644 scripts/kconfig/tests/preprocess/variable/__init__.py

>  create mode 100644 scripts/kconfig/tests/preprocess/variable/expected_stderr

>

> --

> 2.7.4

>




-- 
Best Regards
Masahiro Yamada
Kees Cook April 13, 2018, 5:52 a.m. | #2
On Thu, Apr 12, 2018 at 10:06 PM, Masahiro Yamada
<yamada.masahiro@socionext.com> wrote:
> [Major Changes in V3]


Awesome work! I don't see this pushed to your git tree? I'd like to
test it, but I'd rather "git fetch" instead of "git am" :)

-Kees

-- 
Kees Cook
Pixel Security
Masahiro Yamada April 13, 2018, 12:21 p.m. | #3
2018-04-13 14:52 GMT+09:00 Kees Cook <keescook@chromium.org>:
> On Thu, Apr 12, 2018 at 10:06 PM, Masahiro Yamada

> <yamada.masahiro@socionext.com> wrote:

>> [Major Changes in V3]

>

> Awesome work! I don't see this pushed to your git tree? I'd like to

> test it, but I'd rather "git fetch" instead of "git am" :)

>

> -Kees

>


I pushed this series to the following branch.

git://git.kernel.org/pub/scm/linux/kernel/git/masahiroy/linux-kbuild.git
kconfig-shell-v3



-- 
Best Regards
Masahiro Yamada
Masahiro Yamada April 13, 2018, 1:55 p.m. | #4
2018-04-13 21:21 GMT+09:00 Masahiro Yamada <yamada.masahiro@socionext.com>:
> 2018-04-13 14:52 GMT+09:00 Kees Cook <keescook@chromium.org>:

>> On Thu, Apr 12, 2018 at 10:06 PM, Masahiro Yamada

>> <yamada.masahiro@socionext.com> wrote:

>>> [Major Changes in V3]

>>

>> Awesome work! I don't see this pushed to your git tree? I'd like to

>> test it, but I'd rather "git fetch" instead of "git am" :)

>>

>> -Kees

>>

>

> I pushed this series to the following branch.

>

> git://git.kernel.org/pub/scm/linux/kernel/git/masahiroy/linux-kbuild.git

> kconfig-shell-v3

>



If this approach is successful, we will move more and more
compiler option tests to the Kconfig stage in the future.

People (including me) might be worried about
how slow Kconfig will become.



First, I compared the before/after on my PC.


Without this series,

masahiro@grover:~/workspace/linux-kbuild$ time make -s defconfig

real 0m0.175s
user 0m0.128s
sys 0m0.008s



With this series,
masahiro@grover:~/workspace/linux-kbuild$ time make -s defconfig

real 0m0.729s
user 0m0.400s
sys 0m0.056s


This is noticeable difference.


Then, I looked into per-commit analysis.

Here is the result of the real time of 'time make -s defconfig'

[30/30] kbuild: test dead code/data elimination...     0m0.719s
[29/30] arm64: move GCC version check for...           0m0.711s
[28/30] gcc-plugins: allow to enable GCC_PLUGINS...    0m0.722s
[27/30] gcc-plugins: test plugin support in...         0m0.719s   [+0.31]
[26/30] gcc-plugins: move GCC version check...         0m0.410s
[25/30] kcov: test compiler capability in...           0m0.392s
[24/30] gcov: remove CONFIG_GCOV_FORMAT_AUTODETECT     0m0.400s
[23/30] kconfig: add CC_IS_CLANG and CLANG_VERSION     0m0.396s
[22/30] kconfig: add CC_IS_GCC and GCC_VERSION         0m0.392s
[21/30] stack-protector: test compiler capability...   0m0.381s   [+0.04]
[20/30] kconfig: add basic helper macros to...         0m0.343s
[19/30] kconfig: show compiler version text...         0m0.345s
[18/30] kconfig: test: test text expansion...          0m0.342s
[17/30] Documentation: kconfig: document...            0m0.344s
[16/30] kconfig: add 'info' and 'warning'...           0m0.347s
[15/30] kconfig: expand lefthand side of...            0m0.340s
[14/30] kconfig: support append assignment...          0m0.342s
[13/30] kconfig: support simply expanded...            0m0.341s
[12/30] kconfig: support variable and...               0m0.344s
[11/30] kconfig: begin PARAM state only...             0m0.342s
[10/30] kconfig: replace $(UNAME_RELEASE)...           0m0.347s
[09/30] kconfig: add 'shell' built-in function         0m0.344s
[08/30] kconfig: add built-in function support         0m0.350s
[07/30] kconfig: remove sym_expand_string_value()      0m0.344s
[06/30] kconfig: remove string expansion...            0m0.349s
[05/30] kconfig: remove string expansion...            0m0.342s
[04/30] kconfig: reference environment...              0m0.342s
[03/30] kbuild: remove CONFIG_CROSS_COMPILE...         0m0.347s
[02/30] kbuild: remove kbuild cache                    0m0.347s  [+0.17]
[01/30] gcc-plugins: fix build condition...            0m0.171s
[00/30] Merge tag 'drm-fixes-for-v4.17-rc1'...         0m0.176s


There are three big jump points.

The first one is [02/30]  (+0.17)
We are removing the build cache, so this is what we expect.


The second one is  [21/30]  (+0.04)
For x86, Kconfig runs scripts/gcc-x86_{32,64}-has-stack-protector.sh


The biggest one is [27/30]  (+0.31)
scripts/gcc-plugins.sh is probably very costly script.
If we bump the minimum gcc version to GCC 4.8
the script will be much cleaner in the future.


I was also interested in the cost of
a single $(cc-option ...) invocation.

It is pretty easy to measure this.


For example, copy $(cc-option -fstack-protector)
1000 lines like follows.

config FOO
       bool
       default $(cc-option -fstack-protector)
       default $(cc-option -fstack-protector)
       default $(cc-option -fstack-protector)
       default $(cc-option -fstack-protector)
       default $(cc-option -fstack-protector)
       default $(cc-option -fstack-protector)
       default $(cc-option -fstack-protector)
       default $(cc-option -fstack-protector)
       default $(cc-option -fstack-protector)
       default $(cc-option -fstack-protector)
         ...  [ repeat 1000 line ]



On my core i7 PC, it took 7.2 msec
to run $(cc-option -fstack-protector) 1000 times.


We can make it much faster.

Currently we use
   $(CC) -Werror $(1) -c -x c /dev/null
to test the compiler flag.


Ulf Magnusson suggested to use -S instead of -c
(https://patchwork.kernel.org/patch/10309297/)
With -S, the compiler stops after the compilation stage.
It took only 4.0 msec
to run $(cc-option -fstack-protector) 1000 times


If I use -E  (only pre-process stage), it becomes even faster.
It took only 2.6 msec.


As for $(cc-option ...), probably this will not be a problem.

For some feature, we need special shell-scripts,
some of which can be more costly.




-- 
Best Regards
Masahiro Yamada