cmake: Displace VERBOSE setting

Message ID 20191218233928.696037-1-raj.khem@gmail.com
State New
Headers show
Series
  • cmake: Displace VERBOSE setting
Related show

Commit Message

Khem Raj Dec. 18, 2019, 11:39 p.m.
Setting VERBOSE always, while is fine on one hand for debugging, its
coming at an expense of creating lots and lots of logs, e.g. qtwebkit
compile logs alone with VERBOSE is 163MB, there are many other large
packages which use cmake e.g. WPE, webkitgtk etc which are in same range
with out this option on, the logs reduce to 861K and also speeds up
build a notch

If user needs to enable this logs for targetted debugging debugging that
could be added via

EXTRA_OECMAKE_BUILD += "--verbose"

in recipe

Signed-off-by: Khem Raj <raj.khem@gmail.com>

Cc: Ross Burton <ross.burton@intel.com>
---
 meta/classes/cmake.bbclass | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

-- 
2.24.1

-- 
_______________________________________________
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Comments

André Draszik Dec. 19, 2019, 8:52 a.m. | #1
Hi Khem,

On Wed, 2019-12-18 at 15:39 -0800, Khem Raj wrote:
> Setting VERBOSE always, while is fine on one hand for debugging, its

> coming at an expense of creating lots and lots of logs, e.g. qtwebkit

> compile logs alone with VERBOSE is 163MB, there are many other large

> packages which use cmake e.g. WPE, webkitgtk etc which are in same range

> with out this option on, the logs reduce to 861K and also speeds up

> build a notch

> 

> If user needs to enable this logs for targetted debugging debugging that

> could be added via

> 

> EXTRA_OECMAKE_BUILD += "--verbose"

> 

> in recipe

> 

> Signed-off-by: Khem Raj <raj.khem@gmail.com>

> Cc: Ross Burton <ross.burton@intel.com>


I don't agree with the reasoning to always disable this by default. - it's
way too useful in general. I can't count the number of times it was enough
to look at the log.do_compile to figure out something is wrong.

Now you have to recompile everything with this merged, and something that
took 5 minutes to debug becomes an arduous task.

If the Webkit build is too verbose, you should make that specific recipe
less verbose. E.g. you could always set
EXTRA_OECMAKE_BUILD, and just clear
it out in the webkit recipe.

Otherwise, based on the same reasoning, you should disable all logs, not just
cmake logs: kernel, make output in general, bitbake logs. They all take up space.

Seriously, those logs are useful, keeping in mind that often logs are
collected on build-machines, and having easy access is a huge time-saver.


Cheers,
Andre'



> ---

>  meta/classes/cmake.bbclass | 9 +++++++--

>  1 file changed, 7 insertions(+), 2 deletions(-)

> 

> diff --git a/meta/classes/cmake.bbclass b/meta/classes/cmake.bbclass

> index 8ccb1eefc7..8a44a0c216 100644

> --- a/meta/classes/cmake.bbclass

> +++ b/meta/classes/cmake.bbclass

> @@ -181,9 +181,14 @@ cmake_do_configure() {

>  	  -Wno-dev

>  }

>  

> +# To enable VERBOSE option to emit all kind of logs for a given recipe add following in it

> +#

> +# EXTRA_OECMAKE_BUILD += "--verbose"

> +#

> +# Then run do_compile again

>  cmake_runcmake_build() {

> -	bbnote ${DESTDIR:+DESTDIR=${DESTDIR} }VERBOSE=1 cmake --build '${B}' "$@" -- ${EXTRA_OECMAKE_BUILD}

> -	eval ${DESTDIR:+DESTDIR=${DESTDIR} }VERBOSE=1 cmake --build '${B}' "$@" -- ${EXTRA_OECMAKE_BUILD}

> +	bbnote ${DESTDIR:+DESTDIR=${DESTDIR} } cmake --build '${B}' "$@" -- ${EXTRA_OECMAKE_BUILD}

> +	eval ${DESTDIR:+DESTDIR=${DESTDIR} } cmake --build '${B}' "$@" -- ${EXTRA_OECMAKE_BUILD}

>  }

>  

>  cmake_do_compile()  {

> -- 

> 2.24.1

> 


-- 
_______________________________________________
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Ross Burton Dec. 19, 2019, 12:01 p.m. | #2
On 19/12/2019 08:52, André Draszik wrote:
> Hi Khem,
> 
> On Wed, 2019-12-18 at 15:39 -0800, Khem Raj wrote:
>> Setting VERBOSE always, while is fine on one hand for debugging, its
>> coming at an expense of creating lots and lots of logs, e.g. qtwebkit
>> compile logs alone with VERBOSE is 163MB, there are many other large
>> packages which use cmake e.g. WPE, webkitgtk etc which are in same range
>> with out this option on, the logs reduce to 861K and also speeds up
>> build a notch
>>
>> If user needs to enable this logs for targetted debugging debugging that
>> could be added via
>>
>> EXTRA_OECMAKE_BUILD += "--verbose"
>>
>> in recipe
>>
>> Signed-off-by: Khem Raj <raj.khem@gmail.com>
>> Cc: Ross Burton <ross.burton@intel.com>
> 
> I don't agree with the reasoning to always disable this by default. - it's
> way too useful in general. I can't count the number of times it was enough
> to look at the log.do_compile to figure out something is wrong.
> 
> Now you have to recompile everything with this merged, and something that
> took 5 minutes to debug becomes an arduous task.
> 
> If the Webkit build is too verbose, you should make that specific recipe
> less verbose. E.g. you could always set
> EXTRA_OECMAKE_BUILD, and just clear
> it out in the webkit recipe.
> 
> Otherwise, based on the same reasoning, you should disable all logs, not just
> cmake logs: kernel, make output in general, bitbake logs. They all take up space.
> 
> Seriously, those logs are useful, keeping in mind that often logs are
> collected on build-machines, and having easy access is a huge time-saver.

Personally I agree with André.  We have verbose logs for autotools and 
meson, so why is cmake special.  As this is a webkit-specific issue for 
you, the neater fix would be to override this in webkit.

Stepping back though, why are 163MB log files a problem?  Are you 
archiving every log maybe?  If so then there's an argument to be made 
for a global "logs should be verbose" toggle, that defaults to on but 
you can set to off.  That way the kernel, autotools, etc can also follow 
the same logic and *all* logs can be smaller.

Ross
Khem Raj Dec. 19, 2019, 4:35 p.m. | #3
On Thu, 2019-12-19 at 08:52 +0000, André Draszik wrote:
> Hi Khem,
> 
> On Wed, 2019-12-18 at 15:39 -0800, Khem Raj wrote:
> > Setting VERBOSE always, while is fine on one hand for debugging,
> > its
> > coming at an expense of creating lots and lots of logs, e.g.
> > qtwebkit
> > compile logs alone with VERBOSE is 163MB, there are many other
> > large
> > packages which use cmake e.g. WPE, webkitgtk etc which are in same
> > range
> > with out this option on, the logs reduce to 861K and also speeds up
> > build a notch
> > 
> > If user needs to enable this logs for targetted debugging debugging
> > that
> > could be added via
> > 
> > EXTRA_OECMAKE_BUILD += "--verbose"
> > 
> > in recipe
> > 
> > Signed-off-by: Khem Raj <raj.khem@gmail.com>
> > Cc: Ross Burton <ross.burton@intel.com>
> 
> I don't agree with the reasoning to always disable this by default. -
> it's
> way too useful in general. I can't count the number of times it was
> enough
> to look at the log.do_compile to figure out something is wrong.
> 

What details do you look into when builds dont break ? and how often?
perhaps that will help in understanding the case.

> Now you have to recompile everything with this merged, and something
> that
> took 5 minutes to debug becomes an arduous task.
> 

there are two aspects to it, one you described and another one is where
users and infra is just building it. They are not component developers
rather consumers. I think latter is more common by far.

> If the Webkit build is too verbose, you should make that specific
> recipe
> less verbose. E.g. you could always set
> EXTRA_OECMAKE_BUILD, and just clear
> it out in the webkit recipe.
> 

perhaps that is doable and after current patch it can be set globally
too. However, for an OE user its a bad experience

1. its not default cmake option to enable verbosity, so when someone   
builds their stuff with OE, they don't get good experience, I consider
that added impediment to use OE.

2. People who enable it know what they are doing.

> Otherwise, based on the same reasoning, you should disable all logs,
> not just
> cmake logs: kernel, make output in general, bitbake logs. They all
> take up space.
> 

if they are large perhaps they should be too.
163M is a lot of logs, with slow disks it degrades the compile time,
helps killing the disks quicker, all post processing it delayed, you
pay high price for something you will occasionally use.

> Seriously, those logs are useful, keeping in mind that often logs are
> collected on build-machines, and having easy access is a huge time-
> saver.
> 

One case where people look into these logs are when something breaks
and all error messages are logged without this option as well. 

> 
> Cheers,
> Andre'
> 
> 
> 
> > ---
> >  meta/classes/cmake.bbclass | 9 +++++++--
> >  1 file changed, 7 insertions(+), 2 deletions(-)
> > 
> > diff --git a/meta/classes/cmake.bbclass
> > b/meta/classes/cmake.bbclass
> > index 8ccb1eefc7..8a44a0c216 100644
> > --- a/meta/classes/cmake.bbclass
> > +++ b/meta/classes/cmake.bbclass
> > @@ -181,9 +181,14 @@ cmake_do_configure() {
> >  	  -Wno-dev
> >  }
> >  
> > +# To enable VERBOSE option to emit all kind of logs for a given
> > recipe add following in it
> > +#
> > +# EXTRA_OECMAKE_BUILD += "--verbose"
> > +#
> > +# Then run do_compile again
> >  cmake_runcmake_build() {
> > -	bbnote ${DESTDIR:+DESTDIR=${DESTDIR} }VERBOSE=1 cmake --build
> > '${B}' "$@" -- ${EXTRA_OECMAKE_BUILD}
> > -	eval ${DESTDIR:+DESTDIR=${DESTDIR} }VERBOSE=1 cmake --build
> > '${B}' "$@" -- ${EXTRA_OECMAKE_BUILD}
> > +	bbnote ${DESTDIR:+DESTDIR=${DESTDIR} } cmake --build '${B}'
> > "$@" -- ${EXTRA_OECMAKE_BUILD}
> > +	eval ${DESTDIR:+DESTDIR=${DESTDIR} } cmake --build '${B}' "$@"
> > -- ${EXTRA_OECMAKE_BUILD}
> >  }
> >  
> >  cmake_do_compile()  {
> > -- 
> > 2.24.1
> >
Khem Raj Dec. 19, 2019, 4:41 p.m. | #4
On Thu, 2019-12-19 at 12:01 +0000, Ross Burton wrote:
> On 19/12/2019 08:52, André Draszik wrote:
> > Hi Khem,
> > 
> > On Wed, 2019-12-18 at 15:39 -0800, Khem Raj wrote:
> > > Setting VERBOSE always, while is fine on one hand for debugging,
> > > its
> > > coming at an expense of creating lots and lots of logs, e.g.
> > > qtwebkit
> > > compile logs alone with VERBOSE is 163MB, there are many other
> > > large
> > > packages which use cmake e.g. WPE, webkitgtk etc which are in
> > > same range
> > > with out this option on, the logs reduce to 861K and also speeds
> > > up
> > > build a notch
> > > 
> > > If user needs to enable this logs for targetted debugging
> > > debugging that
> > > could be added via
> > > 
> > > EXTRA_OECMAKE_BUILD += "--verbose"
> > > 
> > > in recipe
> > > 
> > > Signed-off-by: Khem Raj <raj.khem@gmail.com>
> > > Cc: Ross Burton <ross.burton@intel.com>
> > 
> > I don't agree with the reasoning to always disable this by default.
> > - it's
> > way too useful in general. I can't count the number of times it was
> > enough
> > to look at the log.do_compile to figure out something is wrong.
> > 
> > Now you have to recompile everything with this merged, and
> > something that
> > took 5 minutes to debug becomes an arduous task.
> > 
> > If the Webkit build is too verbose, you should make that specific
> > recipe
> > less verbose. E.g. you could always set
> > EXTRA_OECMAKE_BUILD, and just clear
> > it out in the webkit recipe.
> > 
> > Otherwise, based on the same reasoning, you should disable all
> > logs, not just
> > cmake logs: kernel, make output in general, bitbake logs. They all
> > take up space.
> > 
> > Seriously, those logs are useful, keeping in mind that often logs
> > are
> > collected on build-machines, and having easy access is a huge time-
> > saver.
> 
> Personally I agree with André.  We have verbose logs for autotools
> and 
> meson, so why is cmake special.  As this is a webkit-specific issue
> for 
> you, the neater fix would be to override this in webkit.
> 

perhaps there should be a global production mode and curb these logs
too. Since now we are citing webkit as an example solving it just for
webkit does'nt seem right option.

> Stepping back though, why are 163MB log files a problem?  Are you 
> archiving every log maybe?  If so then there's an argument to be
> made 
> for a global "logs should be verbose" toggle, that defaults to on
> but 
> you can set to off.  That way the kernel, autotools, etc can also
> follow 
> the same logic and *all* logs can be smaller.

I agree on this point. Although I wonder if 80% problems can be solved
by fixing cmake verbosity then that not a bad start. After this patch
one could infact enable it globally too by setting it in config
metadata, so we are just toggling defaults. 

> 
> Ross
>

Patch

diff --git a/meta/classes/cmake.bbclass b/meta/classes/cmake.bbclass
index 8ccb1eefc7..8a44a0c216 100644
--- a/meta/classes/cmake.bbclass
+++ b/meta/classes/cmake.bbclass
@@ -181,9 +181,14 @@  cmake_do_configure() {
 	  -Wno-dev
 }
 
+# To enable VERBOSE option to emit all kind of logs for a given recipe add following in it
+#
+# EXTRA_OECMAKE_BUILD += "--verbose"
+#
+# Then run do_compile again
 cmake_runcmake_build() {
-	bbnote ${DESTDIR:+DESTDIR=${DESTDIR} }VERBOSE=1 cmake --build '${B}' "$@" -- ${EXTRA_OECMAKE_BUILD}
-	eval ${DESTDIR:+DESTDIR=${DESTDIR} }VERBOSE=1 cmake --build '${B}' "$@" -- ${EXTRA_OECMAKE_BUILD}
+	bbnote ${DESTDIR:+DESTDIR=${DESTDIR} } cmake --build '${B}' "$@" -- ${EXTRA_OECMAKE_BUILD}
+	eval ${DESTDIR:+DESTDIR=${DESTDIR} } cmake --build '${B}' "$@" -- ${EXTRA_OECMAKE_BUILD}
 }
 
 cmake_do_compile()  {