| Message ID | 20230202175903.2119702-1-alex.bennee@linaro.org |
|---|---|
| State | New |
| Headers | show |
| Series | [RFC] gitlab: call ninja directly and reduce build noise | expand |
Alex Bennée <alex.bennee@linaro.org> writes: > A significant portion of our CI logs are just enumerating each > successfully built object file. The current widespread versions of > ninja don't have a quiet option so we use NINJA_STATUS to add a fixed > string to the ninja output which we then filter with fgrep. If there > are any errors in the output we get them from the compiler. > > Signed-off-by: Alex Bennée <alex.bennee@linaro.org> > --- > .gitlab-ci.d/buildtest-template.yml | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/.gitlab-ci.d/buildtest-template.yml b/.gitlab-ci.d/buildtest-template.yml > index 73ecfabb8d..3c4b237e4f 100644 > --- a/.gitlab-ci.d/buildtest-template.yml > +++ b/.gitlab-ci.d/buildtest-template.yml > @@ -21,7 +21,7 @@ > then > ../meson/meson.py configure . -Dbackend_max_links="$LD_JOBS" ; > fi || exit 1; > - - make -j"$JOBS" > + - env NINJA_STATUS="[ninja][%f/%t] " ninja | fgrep -v "[ninja]" > - if test -n "$MAKE_CHECK_ARGS"; > then > make -j"$JOBS" $MAKE_CHECK_ARGS ; This is too much for gitlab as it trips up on no output at all. What we really need is something that filters ninja output, counts the lines and outputs a dot every 10? 20? lines to keep gitlab happy. Shonky shell script? Very clever sed?
On 2/2/23 08:59, Alex Bennée wrote: >> + - env NINJA_STATUS="[ninja][%f/%t] " ninja | fgrep -v "[ninja]" >> - if test -n "$MAKE_CHECK_ARGS"; >> then >> make -j"$JOBS" $MAKE_CHECK_ARGS ; > > > This is too much for gitlab as it trips up on no output at all. What we > really need is something that filters ninja output, counts the lines and > outputs a dot every 10? 20? lines to keep gitlab happy. > > Shonky shell script? Very clever sed? NINJA_STATUS='ninja %f: ' ninja | grep 'ninja.*[02468]0:' r~
On 02/02/2023 19.59, Alex Bennée wrote: > > Alex Bennée <alex.bennee@linaro.org> writes: > >> A significant portion of our CI logs are just enumerating each >> successfully built object file. The current widespread versions of >> ninja don't have a quiet option so we use NINJA_STATUS to add a fixed >> string to the ninja output which we then filter with fgrep. If there >> are any errors in the output we get them from the compiler. >> >> Signed-off-by: Alex Bennée <alex.bennee@linaro.org> >> --- >> .gitlab-ci.d/buildtest-template.yml | 2 +- >> 1 file changed, 1 insertion(+), 1 deletion(-)0 >> >> diff --git a/.gitlab-ci.d/buildtest-template.yml b/.gitlab-ci.d/buildtest-template.yml >> index 73ecfabb8d..3c4b237e4f 100644 >> --- a/.gitlab-ci.d/buildtest-template.yml >> +++ b/.gitlab-ci.d/buildtest-template.yml >> @@ -21,7 +21,7 @@ >> then >> ../meson/meson.py configure . -Dbackend_max_links="$LD_JOBS" ; >> fi || exit 1; >> - - make -j"$JOBS" >> + - env NINJA_STATUS="[ninja][%f/%t] " ninja | fgrep -v "[ninja]" >> - if test -n "$MAKE_CHECK_ARGS"; >> then >> make -j"$JOBS" $MAKE_CHECK_ARGS ; > > > This is too much for gitlab as it trips up on no output at all. I'm also not sure whether it is really safe to not run make here at all. Some parts of our build system still rely on the "make" magic, I think, and you might miss them if only running ninja. For example pc-bios/optionrom/ does not get compiled without running "make". TBH, I also think the output of the file that currently gets compiled is still valuable in some cases, too, e.g. if you want to be sure whether certain files get compiled in one configuration at all or not. So I'm rather in favor in not doing this change. Thomas
On 3/2/23 08:58, Thomas Huth wrote: > On 02/02/2023 19.59, Alex Bennée wrote: >> >> Alex Bennée <alex.bennee@linaro.org> writes: >> >>> A significant portion of our CI logs are just enumerating each >>> successfully built object file. The current widespread versions of >>> ninja don't have a quiet option so we use NINJA_STATUS to add a fixed >>> string to the ninja output which we then filter with fgrep. If there >>> are any errors in the output we get them from the compiler. >>> >>> Signed-off-by: Alex Bennée <alex.bennee@linaro.org> >>> --- >>> .gitlab-ci.d/buildtest-template.yml | 2 +- >>> 1 file changed, 1 insertion(+), 1 deletion(-)0 >>> >>> diff --git a/.gitlab-ci.d/buildtest-template.yml >>> b/.gitlab-ci.d/buildtest-template.yml >>> index 73ecfabb8d..3c4b237e4f 100644 >>> --- a/.gitlab-ci.d/buildtest-template.yml >>> +++ b/.gitlab-ci.d/buildtest-template.yml >>> @@ -21,7 +21,7 @@ >>> then >>> ../meson/meson.py configure . -Dbackend_max_links="$LD_JOBS" ; >>> fi || exit 1; >>> - - make -j"$JOBS" >>> + - env NINJA_STATUS="[ninja][%f/%t] " ninja | fgrep -v "[ninja]" >>> - if test -n "$MAKE_CHECK_ARGS"; >>> then >>> make -j"$JOBS" $MAKE_CHECK_ARGS ; >> >> >> This is too much for gitlab as it trips up on no output at all. > > I'm also not sure whether it is really safe to not run make here at all. > Some parts of our build system still rely on the "make" magic, I think, > and you might miss them if only running ninja. For example > pc-bios/optionrom/ does not get compiled without running "make". > TBH, I also think the output of the file that currently gets compiled is > still valuable in some cases, too, e.g. if you want to be sure whether > certain files get compiled in one configuration at all or not. So I'm > rather in favor in not doing this change. FWIW I kludge that by using 'make print-x && ninja'...
Richard Henderson <richard.henderson@linaro.org> writes: > On 2/2/23 08:59, Alex Bennée wrote: >>> + - env NINJA_STATUS="[ninja][%f/%t] " ninja | fgrep -v "[ninja]" >>> - if test -n "$MAKE_CHECK_ARGS"; >>> then >>> make -j"$JOBS" $MAKE_CHECK_ARGS ; >> This is too much for gitlab as it trips up on no output at all. What >> we >> really need is something that filters ninja output, counts the lines and >> outputs a dot every 10? 20? lines to keep gitlab happy. >> Shonky shell script? Very clever sed? > > NINJA_STATUS='ninja %f: ' ninja | grep 'ninja.*[02468]0:' NINJA_STATUS="[ninja][%f/%t] " ninja | grep -v "\[ninja\]\[.*[123456789]/" but I guess I need to include it from the Makefile invocation
Thomas Huth <thuth@redhat.com> writes: > On 02/02/2023 19.59, Alex Bennée wrote: >> Alex Bennée <alex.bennee@linaro.org> writes: >> >>> A significant portion of our CI logs are just enumerating each >>> successfully built object file. The current widespread versions of >>> ninja don't have a quiet option so we use NINJA_STATUS to add a fixed >>> string to the ninja output which we then filter with fgrep. If there >>> are any errors in the output we get them from the compiler. >>> >>> Signed-off-by: Alex Bennée <alex.bennee@linaro.org> >>> --- >>> .gitlab-ci.d/buildtest-template.yml | 2 +- >>> 1 file changed, 1 insertion(+), 1 deletion(-)0 >>> >>> diff --git a/.gitlab-ci.d/buildtest-template.yml b/.gitlab-ci.d/buildtest-template.yml >>> index 73ecfabb8d..3c4b237e4f 100644 >>> --- a/.gitlab-ci.d/buildtest-template.yml >>> +++ b/.gitlab-ci.d/buildtest-template.yml >>> @@ -21,7 +21,7 @@ >>> then >>> ../meson/meson.py configure . -Dbackend_max_links="$LD_JOBS" ; >>> fi || exit 1; >>> - - make -j"$JOBS" >>> + - env NINJA_STATUS="[ninja][%f/%t] " ninja | fgrep -v "[ninja]" >>> - if test -n "$MAKE_CHECK_ARGS"; >>> then >>> make -j"$JOBS" $MAKE_CHECK_ARGS ; >> This is too much for gitlab as it trips up on no output at all. > > I'm also not sure whether it is really safe to not run make here at > all. Some parts of our build system still rely on the "make" magic, I > think, and you might miss them if only running ninja. For example > pc-bios/optionrom/ does not get compiled without running "make". OK I'll see if I can embed it in make somehow. > TBH, I also think the output of the file that currently gets compiled > is still valuable in some cases, too, e.g. if you want to be sure > whether certain files get compiled in one configuration at all or not. > So I'm rather in favor in not doing this change. For a successful build you can always go an examine the artefacts after the fact. This is about making the CI log as short and informative as possible to make spotting where errors occur easier. > > Thomas
diff --git a/.gitlab-ci.d/buildtest-template.yml b/.gitlab-ci.d/buildtest-template.yml index 73ecfabb8d..3c4b237e4f 100644 --- a/.gitlab-ci.d/buildtest-template.yml +++ b/.gitlab-ci.d/buildtest-template.yml @@ -21,7 +21,7 @@ then ../meson/meson.py configure . -Dbackend_max_links="$LD_JOBS" ; fi || exit 1; - - make -j"$JOBS" + - env NINJA_STATUS="[ninja][%f/%t] " ninja | fgrep -v "[ninja]" - if test -n "$MAKE_CHECK_ARGS"; then make -j"$JOBS" $MAKE_CHECK_ARGS ;
A significant portion of our CI logs are just enumerating each successfully built object file. The current widespread versions of ninja don't have a quiet option so we use NINJA_STATUS to add a fixed string to the ninja output which we then filter with fgrep. If there are any errors in the output we get them from the compiler. Signed-off-by: Alex Bennée <alex.bennee@linaro.org> --- .gitlab-ci.d/buildtest-template.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)