[RFC] cmake: use Ninja backend instead of Make

Message ID 20180116132559.27794-1-ross.burton@intel.com
State New
Headers show
Series
  • [RFC] cmake: use Ninja backend instead of Make
Related show

Commit Message

Ross Burton Jan. 16, 2018, 1:25 p.m.
The Ninja backend is faster than Make for large projects (takes three minutes
off webkitgtk:do_compile for me), so use it by default.

Signed-off-by: Ross Burton <ross.burton@intel.com>

---
 meta/classes/cmake.bbclass | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

-- 
2.11.0

-- 
_______________________________________________
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Comments

Alexander Kanavin Jan. 16, 2018, 1:29 p.m. | #1
On 01/16/2018 03:25 PM, Ross Burton wrote:
> The Ninja backend is faster than Make for large projects (takes three minutes

> off webkitgtk:do_compile for me), so use it by default.


You do know why Ninja was invented in the first place, right? :)

http://neugierig.org/software/chromium/notes/2011/02/ninja.html

Alex
-- 
_______________________________________________
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Otavio Salvador Jan. 17, 2018, 4:18 p.m. | #2
On Tue, Jan 16, 2018 at 11:25 AM, Ross Burton <ross.burton@intel.com> wrote:
> The Ninja backend is faster than Make for large projects (takes three minutes

> off webkitgtk:do_compile for me), so use it by default.

>

> Signed-off-by: Ross Burton <ross.burton@intel.com>


I tried to test this but it did not apply in master. Could you rebase it?

-- 
Otavio Salvador                             O.S. Systems
http://www.ossystems.com.br        http://code.ossystems.com.br
Mobile: +55 (53) 9981-7854            Mobile: +1 (347) 903-9750
-- 
_______________________________________________
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Randy MacLeod Jan. 17, 2018, 8:57 p.m. | #3
On 2018-01-17 11:18 AM, Otavio Salvador wrote:
> On Tue, Jan 16, 2018 at 11:25 AM, Ross Burton <ross.burton@intel.com> wrote:

>> The Ninja backend is faster than Make for large projects (takes three minutes

>> off webkitgtk:do_compile for me), so use it by default.

>>

>> Signed-off-by: Ross Burton <ross.burton@intel.com>

> 

> I tried to test this but it did not apply in master. Could you rebase it?

> 


On my 64+64 core system that is shared, it reduced a webkit compile
from ~ 17 minutes to ~12 minutes for just the compile stage. YMMV.

$ cat webkit-before-compile.log
3.29user 0.50system 17:39.98elapsed 0%CPU
   (0avgtext+0avgdata 28532maxresident)k
0inputs+176outputs (0major+16335minor)pagefaults 0swaps


$ cat webkit-after-compile.log
2.70user 0.47system 12:09.24elapsed 0%CPU
    (0avgtext+0avgdata 28292maxresident)k
0inputs+160outputs (0major+14634minor)pagefaults 0swaps

I'm going to try to build world (oe-core + meta-oe + ...).


Octavio, it's a two line change! :)
This applied to master for me:

commit 3914c66f6e7123a64ab73e2b6707452bbfd10d02
Author: Ross Burton <ross.burton@intel.com>
Date:   Wed Jan 17 13:09:40 2018 -0500

     cmake: use Ninja backend instead of Make

     The Ninja backend is faster than Make for large projects
     (it takes three minutes off webkitgtk:do_compile for example)
     so use it by default.

     Acked-by: Randy MacLeod <Randy.MacLeod@windriver.com>

     Signed-off-by: Ross Burton <ross.burton@intel.com>


diff --git a/meta/classes/cmake.bbclass b/meta/classes/cmake.bbclass
index a4079dab0d..c46117acde 100644
--- a/meta/classes/cmake.bbclass
+++ b/meta/classes/cmake.bbclass
@@ -1,7 +1,7 @@
  # Path to the CMake file to process.
  OECMAKE_SOURCEPATH ??= "${S}"

-DEPENDS_prepend = "cmake-native "
+DEPENDS_prepend = "cmake-native ninja-native "
  B = "${WORKDIR}/build"

  # We need to unset CCACHE otherwise cmake gets too confused
@@ -117,7 +117,7 @@ cmake_do_configure() {
  		oecmake_sitefile=
  	fi

-	cmake \
+	cmake -GNinja \
  	  $oecmake_sitefile \
  	  ${OECMAKE_SOURCEPATH} \
  	  -DCMAKE_INSTALL_PREFIX:PATH=${prefix} \

-- 
# Randy MacLeod.  WR Linux
# Wind River an Intel Company
-- 
_______________________________________________
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Ross Burton Jan. 17, 2018, 9:43 p.m. | #4
On 17 January 2018 at 20:57, Randy MacLeod <randy.macleod@windriver.com>
wrote:

> On 2018-01-17 11:18 AM, Otavio Salvador wrote:

>

>> On Tue, Jan 16, 2018 at 11:25 AM, Ross Burton <ross.burton@intel.com>

>> wrote:

>>

>>> The Ninja backend is faster than Make for large projects (takes three

>>> minutes

>>> off webkitgtk:do_compile for me), so use it by default.

>>>

>>> Signed-off-by: Ross Burton <ross.burton@intel.com>

>>>

>>

>> I tried to test this but it did not apply in master. Could you rebase it?

>>

>>

> On my 64+64 core system that is shared, it reduced a webkit compile

> from ~ 17 minutes to ~12 minutes for just the compile stage. YMMV.

>

> $ cat webkit-before-compile.log

> 3.29user 0.50system 17:39.98elapsed 0%CPU

>   (0avgtext+0avgdata 28532maxresident)k

> 0inputs+176outputs (0major+16335minor)pagefaults 0swaps

>

>

> $ cat webkit-after-compile.log

> 2.70user 0.47system 12:09.24elapsed 0%CPU

>    (0avgtext+0avgdata 28292maxresident)k

> 0inputs+160outputs (0major+14634minor)pagefaults 0swaps

>

> I'm going to try to build world (oe-core + meta-oe + ...).

>


Good win :)

There appears to be a small number of upstreams that break with ninja, so
I'll rework this patch so that the backend is an option but default to
Ninja.

Ross
<div dir="ltr">On 17 January 2018 at 20:57, Randy MacLeod <span dir="ltr">&lt;<a href="mailto:randy.macleod@windriver.com" target="_blank">randy.macleod@windriver.com</a>&gt;</span> wrote:<br><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">On 2018-01-17 11:18 AM, Otavio Salvador wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
On Tue, Jan 16, 2018 at 11:25 AM, Ross Burton &lt;<a href="mailto:ross.burton@intel.com" target="_blank">ross.burton@intel.com</a>&gt; wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
The Ninja backend is faster than Make for large projects (takes three minutes<br>
off webkitgtk:do_compile for me), so use it by default.<br>
<br>
Signed-off-by: Ross Burton &lt;<a href="mailto:ross.burton@intel.com" target="_blank">ross.burton@intel.com</a>&gt;<br>

</blockquote>
<br>
I tried to test this but it did not apply in master. Could you rebase it?<br>
<br>
</blockquote>
<br></span>
On my 64+64 core system that is shared, it reduced a webkit compile<br>
from ~ 17 minutes to ~12 minutes for just the compile stage. YMMV.<br>
<br>
$ cat webkit-before-compile.log<br>
3.29user 0.50system 17:39.98elapsed 0%CPU<br>
  (0avgtext+0avgdata 28532maxresident)k<br>
0inputs+176outputs (0major+16335minor)pagefaults 0swaps<br>
<br>
<br>
$ cat webkit-after-compile.log<br>
2.70user 0.47system 12:09.24elapsed 0%CPU<br>
   (0avgtext+0avgdata 28292maxresident)k<br>
0inputs+160outputs (0major+14634minor)pagefaults 0swaps<br>
<br>
I&#39;m going to try to build world (oe-core + meta-oe + ...).<br></blockquote><div><br></div><div>Good win :)</div><div><br></div><div>There appears to be a small number of upstreams that break with ninja, so I&#39;ll rework this patch so that the backend is an option but default to Ninja.</div><div><br></div><div>Ross </div></div></div></div>
-- 
_______________________________________________
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Otavio Salvador Jan. 18, 2018, 12:13 a.m. | #5
On Wed, Jan 17, 2018 at 7:43 PM, Burton, Ross <ross.burton@intel.com> wrote:
> There appears to be a small number of upstreams that break with ninja, so

> I'll rework this patch so that the backend is an option but default to

> Ninja.


When doing so, please rebase on top of my patches and send it as
official patch (no RFC). Besides test, it is very welcome.

-- 
Otavio Salvador                             O.S. Systems
http://www.ossystems.com.br        http://code.ossystems.com.br
Mobile: +55 (53) 9981-7854            Mobile: +1 (347) 903-9750
-- 
_______________________________________________
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Randy MacLeod Jan. 18, 2018, 2:59 a.m. | #6
On 2018-01-17 04:43 PM, Burton, Ross wrote:
> There appears to be a small number of upstreams that break with ninja, 

> so I'll rework this patch so that the backend is an option but default 

> to Ninja.


The only failure that involved a cmake based package was piglit
for my qemux86-64 world build.

It didn't install:
   make: Entering directory 
'/.../tmp-glibc/work/core2-64-wrs-linux/piglit/1.0+gitrAUTOINC+c8f4fd9eeb-r0/build'
make: *** No rule to make target 'install/strip'.  Stop.

so that might not be related to this change. I haven't manually done
a world build on master recently.

As expected using ninja for the back-end of cmake just makes the
builds faster.

-- 
# Randy MacLeod.  WR Linux
# Wind River an Intel Company
-- 
_______________________________________________
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Randy MacLeod Jan. 18, 2018, 3:04 a.m. | #7
On 2018-01-17 09:59 PM, Randy MacLeod wrote:
> On 2018-01-17 04:43 PM, Burton, Ross wrote:
>> There appears to be a small number of upstreams that break with ninja, 
>> so I'll rework this patch so that the backend is an option but default 
>> to Ninja.
> 
> The only failure that involved a cmake based package was piglit
> for my qemux86-64 world build.
> 
> It didn't install:
>    make: Entering directory 
> '/.../tmp-glibc/work/core2-64-wrs-linux/piglit/1.0+gitrAUTOINC+c8f4fd9eeb-r0/build' 
> 
> make: *** No rule to make target 'install/strip'.  Stop.
> 
> so that might not be related to this change. I haven't manually done
> a world build on master recently.

Apparently already fixed by Ross in master-next:

commit 9fec5ef920bd63fe3cc2e623add0c7aead95ecae
Author: Ross Burton <ross.burton@intel.com>
Date:   Tue Jan 16 13:25:16 2018 +0000

     piglit: override install target name instead of do_install

../Randy

> 
> As expected using ninja for the back-end of cmake just makes the
> builds faster.
>

Patch

diff --git a/meta/classes/cmake.bbclass b/meta/classes/cmake.bbclass
index 10c4edcf1a5..ab29b1f1c23 100644
--- a/meta/classes/cmake.bbclass
+++ b/meta/classes/cmake.bbclass
@@ -1,7 +1,7 @@ 
 # Path to the CMake file to process.
 OECMAKE_SOURCEPATH ??= "${S}"
 
-DEPENDS_prepend = "cmake-native "
+DEPENDS_prepend = "cmake-native ninja-native "
 B = "${WORKDIR}/build"
 
 # We need to unset CCACHE otherwise cmake gets too confused
@@ -120,7 +120,7 @@  cmake_do_configure() {
 		oecmake_sitefile=
 	fi
 
-	cmake \
+	cmake -GNinja \
 	  $oecmake_sitefile \
 	  ${OECMAKE_SOURCEPATH} \
 	  -DCMAKE_INSTALL_PREFIX:PATH=${prefix} \
@@ -141,7 +141,7 @@  cmake_do_configure() {
 	  -Wno-dev
 }
 
-do_compile[progress] = "percent"
+do_compile[progress] = "outof:^\[(\d+)/(\d+)\]\s+"
 cmake_do_compile()  {
 	bbnote VERBOSE=1 cmake --build '${B}' --target ${OECMAKE_TARGET_COMPILE} -- ${EXTRA_OECMAKE_BUILD}
 	VERBOSE=1 cmake --build '${B}' --target ${OECMAKE_TARGET_COMPILE} -- ${EXTRA_OECMAKE_BUILD}