diff mbox series

[2/3] kbuild: prepare to remove C files pre-generated by flex and bison

Message ID 1512835350-8646-3-git-send-email-yamada.masahiro@socionext.com
State Accepted
Commit 033dba2ec06c47a9fe1b190bc3281058fb20738d
Headers show
Series kbuild,kconfig: generate lexer/parser C files instead of copying _shipped files | expand

Commit Message

Masahiro Yamada Dec. 9, 2017, 4:02 p.m. UTC 
In Linux build system convention, pre-generated files are version-
controlled with a "_shipped" suffix.  During the kernel building,
they are simply shipped (copied) removing the suffix.

This approach can reduce external tool dependency for the kernel build,
but it is tedious to manually regenerate such artifacts from developers'
point of view.  (We need to do "make REGENERATE_PARSERS=1" every time
we touch real source files such as *.l, *.y)

Some months ago, I sent out RFC patches to run flex, bison, and gperf
during the build.

In the review and test, Linus noticed gperf-3.1 had changed the lookup
function prototype.  Then, the use of gperf in kernel was entirely
removed by commit bb3290d91695 ("Remove gperf usage from toolchain").

I tested several versions of flex and bison, and I was not hit by any
compatibility issue except a flaw in flex-2.6.3; if you generate lexer
for dtc and genksyms with flex-2.6.3, you will see "yywrap redefined"
warning.  This was not intentional, but a bug fixed by flex-2.6.4.
Otherwise, flex and bison look fairly stable for a long time.

This commit prepares some build rules to start removing the _shipped
files.  Also, document minimal requirement for flex and bison.

Rationale for the minimal version:
The -Wmissing-prototypes option of GCC reports "no previous prototype"
warnings for lexers generated by flex-2.5.34, so I chose 2.5.35 as the
required version for flex.  Flex-2.5.35 was released in 2008.  Bison
looked more stable.  I did not see any problem with bison-2.0, released
in 2004.  I did not test bison-1.x, but bison-2.0 should be old enough.

Tested flex versions:
  2.5.35
  2.5.36
  2.5.37
  2.5.39
  2.6.0
  2.6.1
  2.6.2
  2.6.3   (*)
  2.6.4

 (*) flex-2.6.3 causes "yywrap redefined" warning

Tested bison versions:
  2.0
  2.1
  2.2
  2.3
  2.4
  2.4.1
  2.5.1
  2.6
  2.6.1
  2.6.2
  2.6.3
  2.6.4
  2.6.5
  2.7
  2.7.1
  3.0
  3.0.1
  3.0.2
  3.0.3
  3.0.4

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>

---

 Documentation/process/changes.rst | 25 +++++++++++++++++++++++++
 scripts/Makefile.lib              | 20 +++++++++++++++++---
 2 files changed, 42 insertions(+), 3 deletions(-)

-- 
2.7.4

Comments

Randy Dunlap Dec. 9, 2017, 4:47 p.m. UTC | #1 
On 12/09/2017 08:02 AM, Masahiro Yamada wrote:
> 

> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>

> ---

> 

>  Documentation/process/changes.rst | 25 +++++++++++++++++++++++++

>  scripts/Makefile.lib              | 20 +++++++++++++++++---

>  2 files changed, 42 insertions(+), 3 deletions(-)

> 

> diff --git a/Documentation/process/changes.rst b/Documentation/process/changes.rst

> index 560beae..fc9c7c3 100644

> --- a/Documentation/process/changes.rst

> +++ b/Documentation/process/changes.rst

> @@ -32,6 +32,8 @@ you probably needn't concern yourself with isdn4k-utils.

>  GNU C                  3.2              gcc --version

>  GNU make               3.81             make --version

>  binutils               2.20             ld -v

> +flex                   2.5.35           flex --version

> +bison                  2.0              bison --version

>  util-linux             2.10o            fdformat --version

>  module-init-tools      0.9.10           depmod -V

>  e2fsprogs              1.41.4           e2fsck -V

> @@ -79,6 +81,19 @@ The build system has, as of 4.13, switched to using thin archives (`ar T`)

>  rather than incremental linking (`ld -r`) for built-in.o intermediate steps.

>  This requires binutils 2.20 or newer.

>  

> +Flex

> +----

> +

> +Since Linux 4.16, the build system generates lexical analisers


                                                        analyzers
or                                                      analysers

> +during build.  This requires flex 2.5.35 or later.



thanks.
-- 
~Randy
diff mbox series

Patch

diff --git a/Documentation/process/changes.rst b/Documentation/process/changes.rst
index 560beae..fc9c7c3 100644
--- a/Documentation/process/changes.rst
+++ b/Documentation/process/changes.rst
@@ -32,6 +32,8 @@  you probably needn't concern yourself with isdn4k-utils.
 GNU C                  3.2              gcc --version
 GNU make               3.81             make --version
 binutils               2.20             ld -v
+flex                   2.5.35           flex --version
+bison                  2.0              bison --version
 util-linux             2.10o            fdformat --version
 module-init-tools      0.9.10           depmod -V
 e2fsprogs              1.41.4           e2fsck -V
@@ -79,6 +81,19 @@  The build system has, as of 4.13, switched to using thin archives (`ar T`)
 rather than incremental linking (`ld -r`) for built-in.o intermediate steps.
 This requires binutils 2.20 or newer.
 
+Flex
+----
+
+Since Linux 4.16, the build system generates lexical analisers
+during build.  This requires flex 2.5.35 or later.
+
+
+Bison
+-----
+
+Since Linux 4.16, the build system generates parsers
+during build.  This requires bison 2.0 or later.
+
 Perl
 ----
 
@@ -333,6 +348,16 @@  Binutils
 
 - <https://www.kernel.org/pub/linux/devel/binutils/>
 
+Flex
+----
+
+- <https://github.com/westes/flex/releases>
+
+Bison
+-----
+
+- <ftp://ftp.gnu.org/gnu/bison/>
+
 OpenSSL
 -------
 
diff --git a/scripts/Makefile.lib b/scripts/Makefile.lib
index ee528e3..0f9ef3f 100644
--- a/scripts/Makefile.lib
+++ b/scripts/Makefile.lib
@@ -186,8 +186,6 @@  $(foreach m, $(notdir $1), \
 	$(addprefix $(obj)/, $(foreach s, $3, $($(m:%$(strip $2)=%$(s)))))))
 endef
 
-ifdef REGENERATE_PARSERS
-
 # LEX
 # ---------------------------------------------------------------------------
 LEX_PREFIX = $(if $(LEX_PREFIX_${baseprereq}),$(LEX_PREFIX_${baseprereq}),yy)
@@ -195,9 +193,15 @@  LEX_PREFIX = $(if $(LEX_PREFIX_${baseprereq}),$(LEX_PREFIX_${baseprereq}),yy)
 quiet_cmd_flex = LEX     $@
       cmd_flex = $(LEX) -o$@ -L -P $(LEX_PREFIX) $<
 
+ifdef REGENERATE_PARSERS
 .PRECIOUS: $(src)/%.lex.c_shipped
 $(src)/%.lex.c_shipped: $(src)/%.l
 	$(call cmd,flex)
+endif
+
+.PRECIOUS: $(obj)/%.lex.c
+$(filter %.lex.c,$(targets)): $(obj)/%.lex.c: $(src)/%.l FORCE
+	$(call if_changed,flex)
 
 # YACC
 # ---------------------------------------------------------------------------
@@ -206,19 +210,29 @@  YACC_PREFIX = $(if $(YACC_PREFIX_${baseprereq}),$(YACC_PREFIX_${baseprereq}),yy)
 quiet_cmd_bison = YACC    $@
       cmd_bison = $(YACC) -o$@ -t -l -p $(YACC_PREFIX) $<
 
+ifdef REGENERATE_PARSERS
 .PRECIOUS: $(src)/%.tab.c_shipped
 $(src)/%.tab.c_shipped: $(src)/%.y
 	$(call cmd,bison)
+endif
+
+.PRECIOUS: $(obj)/%.tab.c
+$(filter %.tab.c,$(targets)): $(obj)/%.tab.c: $(src)/%.y FORCE
+	$(call if_changed,bison)
 
 quiet_cmd_bison_h = YACC    $@
       cmd_bison_h = bison -o/dev/null --defines=$@ -t -l -p $(YACC_PREFIX) $<
 
+ifdef REGENERATE_PARSERS
 .PRECIOUS: $(src)/%.tab.h_shipped
 $(src)/%.tab.h_shipped: $(src)/%.y
 	$(call cmd,bison_h)
-
 endif
 
+.PRECIOUS: $(obj)/%.tab.h
+$(filter %.tab.h,$(targets)): $(obj)/%.tab.h: $(src)/%.y FORCE
+	$(call if_changed,bison_h)
+
 # Shipped files
 # ===========================================================================