| Message ID | 20250503201806.3045723-1-pierrick.bouvier@linaro.org |
|---|---|
| State | New |
| Headers | show |
| Series | meson: use thorough test setup as default | expand |
Il sab 3 mag 2025, 22:18 Pierrick Bouvier <pierrick.bouvier@linaro.org> ha scritto: > Allows all tests to be visible by default when using meson test > directly. > > This has no impact on make check-* commands, which use SPEED=quick by > default (see scripts/mtest2make.py). > What's the advantage of having different defaults depending on whether you use "make check" or "meson test"? I don't oppose this change per se, but if it's useful it should be matched by a change in the default SPEED. Paolo > Signed-off-by: Pierrick Bouvier <pierrick.bouvier@linaro.org> > --- > meson.build | 5 +++-- > 1 file changed, 3 insertions(+), 2 deletions(-) > > diff --git a/meson.build b/meson.build > index f8bf6e7bb66..57ff3f722d8 100644 > --- a/meson.build > +++ b/meson.build > @@ -5,12 +5,13 @@ project('qemu', ['c'], meson_version: '>=1.5.0', > > meson.add_devenv({ 'MESON_BUILD_ROOT' : meson.project_build_root() }) > > -add_test_setup('quick', exclude_suites: ['slow', 'thorough'], is_default: > true, > +add_test_setup('quick', exclude_suites: ['slow', 'thorough'], > env: ['RUST_BACKTRACE=1']) > add_test_setup('slow', exclude_suites: ['thorough'], > env: ['G_TEST_SLOW=1', 'SPEED=slow', 'RUST_BACKTRACE=1']) > add_test_setup('thorough', > - env: ['G_TEST_SLOW=1', 'SPEED=thorough', > 'RUST_BACKTRACE=1']) > + env: ['G_TEST_SLOW=1', 'SPEED=thorough', > 'RUST_BACKTRACE=1'], > + is_default: true) > > meson.add_postconf_script(find_program('scripts/symlink-install-tree.py')) > > -- > 2.47.2 > >
On 03/05/2025 22.18, Pierrick Bouvier wrote: > Allows all tests to be visible by default when using meson test > directly. > > This has no impact on make check-* commands, which use SPEED=quick by > default (see scripts/mtest2make.py). > > Signed-off-by: Pierrick Bouvier <pierrick.bouvier@linaro.org> > --- > meson.build | 5 +++-- > 1 file changed, 3 insertions(+), 2 deletions(-) > > diff --git a/meson.build b/meson.build > index f8bf6e7bb66..57ff3f722d8 100644 > --- a/meson.build > +++ b/meson.build > @@ -5,12 +5,13 @@ project('qemu', ['c'], meson_version: '>=1.5.0', > > meson.add_devenv({ 'MESON_BUILD_ROOT' : meson.project_build_root() }) > > -add_test_setup('quick', exclude_suites: ['slow', 'thorough'], is_default: true, > +add_test_setup('quick', exclude_suites: ['slow', 'thorough'], > env: ['RUST_BACKTRACE=1']) > add_test_setup('slow', exclude_suites: ['thorough'], > env: ['G_TEST_SLOW=1', 'SPEED=slow', 'RUST_BACKTRACE=1']) > add_test_setup('thorough', > - env: ['G_TEST_SLOW=1', 'SPEED=thorough', 'RUST_BACKTRACE=1']) > + env: ['G_TEST_SLOW=1', 'SPEED=thorough', 'RUST_BACKTRACE=1'], > + is_default: true) I'd rather not make it the default: The thorough functional tests download a lot of assets from the internet, so if someone just runs "meson test" without further parameters, I think we should not trigger these downloads in that case. Thomas
On 5/4/25 11:00 PM, Paolo Bonzini wrote: > > > Il sab 3 mag 2025, 22:18 Pierrick Bouvier <pierrick.bouvier@linaro.org > <mailto:pierrick.bouvier@linaro.org>> ha scritto: > > Allows all tests to be visible by default when using meson test > directly. > > This has no impact on make check-* commands, which use SPEED=quick by > default (see scripts/mtest2make.py). > > > What's the advantage of having different defaults depending on whether > you use "make check" or "meson test"? > "make check" can be seen as a blackbox, where you have different targets, but don't really do things on a per test basis. At least, it's not the workflow that was implemented. "meson test", on the opposite, expose the list of tests to you. My rationale here is to expose all tests for meson test (instead of having to add --setup thorough everytime), and let the "per setup" workflow to make command. > I don't oppose this change per se, but if it's useful it should be > matched by a change in the default SPEED. > As Thomas pointed, running thorough tests by default would take more time/more disk space, and I'm not sure we want to change what is done in our CI or for people by default. This change is just supposed to be convenience for people using meson test directly, without impacting "make check*" users. So when you want to run a single test, it's visible from "meson test -C build --list", by default. > Paolo > > > Signed-off-by: Pierrick Bouvier <pierrick.bouvier@linaro.org > <mailto:pierrick.bouvier@linaro.org>> > --- > meson.build | 5 +++-- > 1 file changed, 3 insertions(+), 2 deletions(-) > > diff --git a/meson.build b/meson.build > index f8bf6e7bb66..57ff3f722d8 100644 > --- a/meson.build > +++ b/meson.build > @@ -5,12 +5,13 @@ project('qemu', ['c'], meson_version: '>=1.5.0', > > meson.add_devenv({ 'MESON_BUILD_ROOT' : meson.project_build_root() }) > > -add_test_setup('quick', exclude_suites: ['slow', 'thorough'], > is_default: true, > +add_test_setup('quick', exclude_suites: ['slow', 'thorough'], > env: ['RUST_BACKTRACE=1']) > add_test_setup('slow', exclude_suites: ['thorough'], > env: ['G_TEST_SLOW=1', 'SPEED=slow', > 'RUST_BACKTRACE=1']) > add_test_setup('thorough', > - env: ['G_TEST_SLOW=1', 'SPEED=thorough', > 'RUST_BACKTRACE=1']) > + env: ['G_TEST_SLOW=1', 'SPEED=thorough', > 'RUST_BACKTRACE=1'], > + is_default: true) > > meson.add_postconf_script(find_program('scripts/symlink-install- > tree.py')) > > -- > 2.47.2 >
On 5/5/25 3:32 AM, Thomas Huth wrote: > On 03/05/2025 22.18, Pierrick Bouvier wrote: >> Allows all tests to be visible by default when using meson test >> directly. >> >> This has no impact on make check-* commands, which use SPEED=quick by >> default (see scripts/mtest2make.py). >> >> Signed-off-by: Pierrick Bouvier <pierrick.bouvier@linaro.org> >> --- >> meson.build | 5 +++-- >> 1 file changed, 3 insertions(+), 2 deletions(-) >> >> diff --git a/meson.build b/meson.build >> index f8bf6e7bb66..57ff3f722d8 100644 >> --- a/meson.build >> +++ b/meson.build >> @@ -5,12 +5,13 @@ project('qemu', ['c'], meson_version: '>=1.5.0', >> >> meson.add_devenv({ 'MESON_BUILD_ROOT' : meson.project_build_root() }) >> >> -add_test_setup('quick', exclude_suites: ['slow', 'thorough'], is_default: true, >> +add_test_setup('quick', exclude_suites: ['slow', 'thorough'], >> env: ['RUST_BACKTRACE=1']) >> add_test_setup('slow', exclude_suites: ['thorough'], >> env: ['G_TEST_SLOW=1', 'SPEED=slow', 'RUST_BACKTRACE=1']) >> add_test_setup('thorough', >> - env: ['G_TEST_SLOW=1', 'SPEED=thorough', 'RUST_BACKTRACE=1']) >> + env: ['G_TEST_SLOW=1', 'SPEED=thorough', 'RUST_BACKTRACE=1'], >> + is_default: true) > > I'd rather not make it the default: The thorough functional tests download a > lot of assets from the internet, so if someone just runs "meson test" > without further parameters, I think we should not trigger these downloads in > that case. > $ cat tests/Makefile.include ... check-functional: @$(NINJA) precache-functional @QEMU_TEST_NO_DOWNLOAD=1 $(MAKE) SPEED=thorough check-func ... What's the rationale to run check-func with SPEED=thorough and QEMU_TEST_NO_DOWNLOAD=1 with precache-functional having cached only quick tests by default? Either we should remove SPEED=thorough, or download all tests by default. > Thomas >
On Mon, May 05, 2025 at 10:46:52AM -0700, Pierrick Bouvier wrote: > On 5/5/25 3:32 AM, Thomas Huth wrote: > > On 03/05/2025 22.18, Pierrick Bouvier wrote: > > > Allows all tests to be visible by default when using meson test > > > directly. > > > > > > This has no impact on make check-* commands, which use SPEED=quick by > > > default (see scripts/mtest2make.py). > > > > > > Signed-off-by: Pierrick Bouvier <pierrick.bouvier@linaro.org> > > > --- > > > meson.build | 5 +++-- > > > 1 file changed, 3 insertions(+), 2 deletions(-) > > > > > > diff --git a/meson.build b/meson.build > > > index f8bf6e7bb66..57ff3f722d8 100644 > > > --- a/meson.build > > > +++ b/meson.build > > > @@ -5,12 +5,13 @@ project('qemu', ['c'], meson_version: '>=1.5.0', > > > meson.add_devenv({ 'MESON_BUILD_ROOT' : meson.project_build_root() }) > > > -add_test_setup('quick', exclude_suites: ['slow', 'thorough'], is_default: true, > > > +add_test_setup('quick', exclude_suites: ['slow', 'thorough'], > > > env: ['RUST_BACKTRACE=1']) > > > add_test_setup('slow', exclude_suites: ['thorough'], > > > env: ['G_TEST_SLOW=1', 'SPEED=slow', 'RUST_BACKTRACE=1']) > > > add_test_setup('thorough', > > > - env: ['G_TEST_SLOW=1', 'SPEED=thorough', 'RUST_BACKTRACE=1']) > > > + env: ['G_TEST_SLOW=1', 'SPEED=thorough', 'RUST_BACKTRACE=1'], > > > + is_default: true) > > > > I'd rather not make it the default: The thorough functional tests download a > > lot of assets from the internet, so if someone just runs "meson test" > > without further parameters, I think we should not trigger these downloads in > > that case. > > > > $ cat tests/Makefile.include > ... > check-functional: > @$(NINJA) precache-functional > @QEMU_TEST_NO_DOWNLOAD=1 $(MAKE) SPEED=thorough check-func > ... > > What's the rationale to run check-func with SPEED=thorough and > QEMU_TEST_NO_DOWNLOAD=1 with precache-functional having cached only quick > tests by default? Are you sure about that ? The precache logic uses custom_target() in meson and so does not filters on "suites" used to define quick vs thorough tests. IOW, precache should always be downloading all assets. > Either we should remove SPEED=thorough, or download all tests by default. > > > Thomas > > > With regards, Daniel
On Sat, May 03, 2025 at 01:18:06PM -0700, Pierrick Bouvier wrote: > Allows all tests to be visible by default when using meson test > directly. > > This has no impact on make check-* commands, which use SPEED=quick by > default (see scripts/mtest2make.py). IMHO it would be conceptually confusing if we cause 'make check' to be running different stuff from 'meson test'. As long as we keep a makefile wrapper around running tests, I think we should keep them matching what they do by default. > > Signed-off-by: Pierrick Bouvier <pierrick.bouvier@linaro.org> > --- > meson.build | 5 +++-- > 1 file changed, 3 insertions(+), 2 deletions(-) > > diff --git a/meson.build b/meson.build > index f8bf6e7bb66..57ff3f722d8 100644 > --- a/meson.build > +++ b/meson.build > @@ -5,12 +5,13 @@ project('qemu', ['c'], meson_version: '>=1.5.0', > > meson.add_devenv({ 'MESON_BUILD_ROOT' : meson.project_build_root() }) > > -add_test_setup('quick', exclude_suites: ['slow', 'thorough'], is_default: true, > +add_test_setup('quick', exclude_suites: ['slow', 'thorough'], > env: ['RUST_BACKTRACE=1']) > add_test_setup('slow', exclude_suites: ['thorough'], > env: ['G_TEST_SLOW=1', 'SPEED=slow', 'RUST_BACKTRACE=1']) > add_test_setup('thorough', > - env: ['G_TEST_SLOW=1', 'SPEED=thorough', 'RUST_BACKTRACE=1']) > + env: ['G_TEST_SLOW=1', 'SPEED=thorough', 'RUST_BACKTRACE=1'], > + is_default: true) > > meson.add_postconf_script(find_program('scripts/symlink-install-tree.py')) > > -- > 2.47.2 > With regards, Daniel
On 06/05/2025 11.36, Daniel P. Berrangé wrote: > On Sat, May 03, 2025 at 01:18:06PM -0700, Pierrick Bouvier wrote: >> Allows all tests to be visible by default when using meson test >> directly. >> >> This has no impact on make check-* commands, which use SPEED=quick by >> default (see scripts/mtest2make.py). > > IMHO it would be conceptually confusing if we cause 'make check' to > be running different stuff from 'meson test'. As long as we keep a > makefile wrapper around running tests, I think we should keep them > matching what they do by default. But what we should maybe do instead: Properly document in docs/devel/testing/functional.rst how to get the list of all tests and how to run a single test with the "meson test" way... I could do it later when time permits (likely not this week), but if anybody else wants to have a try, that would be welcome! Thomas
On 5/6/25 2:33 AM, Daniel P. Berrangé wrote: > On Mon, May 05, 2025 at 10:46:52AM -0700, Pierrick Bouvier wrote: >> On 5/5/25 3:32 AM, Thomas Huth wrote: >>> On 03/05/2025 22.18, Pierrick Bouvier wrote: >>>> Allows all tests to be visible by default when using meson test >>>> directly. >>>> >>>> This has no impact on make check-* commands, which use SPEED=quick by >>>> default (see scripts/mtest2make.py). >>>> >>>> Signed-off-by: Pierrick Bouvier <pierrick.bouvier@linaro.org> >>>> --- >>>> meson.build | 5 +++-- >>>> 1 file changed, 3 insertions(+), 2 deletions(-) >>>> >>>> diff --git a/meson.build b/meson.build >>>> index f8bf6e7bb66..57ff3f722d8 100644 >>>> --- a/meson.build >>>> +++ b/meson.build >>>> @@ -5,12 +5,13 @@ project('qemu', ['c'], meson_version: '>=1.5.0', >>>> meson.add_devenv({ 'MESON_BUILD_ROOT' : meson.project_build_root() }) >>>> -add_test_setup('quick', exclude_suites: ['slow', 'thorough'], is_default: true, >>>> +add_test_setup('quick', exclude_suites: ['slow', 'thorough'], >>>> env: ['RUST_BACKTRACE=1']) >>>> add_test_setup('slow', exclude_suites: ['thorough'], >>>> env: ['G_TEST_SLOW=1', 'SPEED=slow', 'RUST_BACKTRACE=1']) >>>> add_test_setup('thorough', >>>> - env: ['G_TEST_SLOW=1', 'SPEED=thorough', 'RUST_BACKTRACE=1']) >>>> + env: ['G_TEST_SLOW=1', 'SPEED=thorough', 'RUST_BACKTRACE=1'], >>>> + is_default: true) >>> >>> I'd rather not make it the default: The thorough functional tests download a >>> lot of assets from the internet, so if someone just runs "meson test" >>> without further parameters, I think we should not trigger these downloads in >>> that case. >>> >> >> $ cat tests/Makefile.include >> ... >> check-functional: >> @$(NINJA) precache-functional >> @QEMU_TEST_NO_DOWNLOAD=1 $(MAKE) SPEED=thorough check-func >> ... >> >> What's the rationale to run check-func with SPEED=thorough and >> QEMU_TEST_NO_DOWNLOAD=1 with precache-functional having cached only quick >> tests by default? > > Are you sure about that ? The precache logic uses custom_target() in > meson and so does not filters on "suites" used to define quick vs > thorough tests. > My bad, I missed the "foreach speed : ['quick', 'thorough']" wrapping all this in tests/functional/meson.build. > IOW, precache should always be downloading all assets. > Then I don't understand the previous argument from Thomas to not make thorough the default: "The thorough functional tests download a lot of assets from the internet, so if someone just runs "meson test" without further parameters, I think we should not trigger these downloads in that case". It's what precache-functional is doing. >> Either we should remove SPEED=thorough, or download all tests by default. >> >>> Thomas >>> >> > > With regards, > Daniel
On Tue, May 06, 2025 at 08:31:07AM -0700, Pierrick Bouvier wrote: > On 5/6/25 2:33 AM, Daniel P. Berrangé wrote: > > On Mon, May 05, 2025 at 10:46:52AM -0700, Pierrick Bouvier wrote: > > > On 5/5/25 3:32 AM, Thomas Huth wrote: > > > > On 03/05/2025 22.18, Pierrick Bouvier wrote: > > > > > Allows all tests to be visible by default when using meson test > > > > > directly. > > > > > > > > > > This has no impact on make check-* commands, which use SPEED=quick by > > > > > default (see scripts/mtest2make.py). > > > > > > > > > > Signed-off-by: Pierrick Bouvier <pierrick.bouvier@linaro.org> > > > > > --- > > > > > meson.build | 5 +++-- > > > > > 1 file changed, 3 insertions(+), 2 deletions(-) > > > > > > > > > > diff --git a/meson.build b/meson.build > > > > > index f8bf6e7bb66..57ff3f722d8 100644 > > > > > --- a/meson.build > > > > > +++ b/meson.build > > > > > @@ -5,12 +5,13 @@ project('qemu', ['c'], meson_version: '>=1.5.0', > > > > > meson.add_devenv({ 'MESON_BUILD_ROOT' : meson.project_build_root() }) > > > > > -add_test_setup('quick', exclude_suites: ['slow', 'thorough'], is_default: true, > > > > > +add_test_setup('quick', exclude_suites: ['slow', 'thorough'], > > > > > env: ['RUST_BACKTRACE=1']) > > > > > add_test_setup('slow', exclude_suites: ['thorough'], > > > > > env: ['G_TEST_SLOW=1', 'SPEED=slow', 'RUST_BACKTRACE=1']) > > > > > add_test_setup('thorough', > > > > > - env: ['G_TEST_SLOW=1', 'SPEED=thorough', 'RUST_BACKTRACE=1']) > > > > > + env: ['G_TEST_SLOW=1', 'SPEED=thorough', 'RUST_BACKTRACE=1'], > > > > > + is_default: true) > > > > > > > > I'd rather not make it the default: The thorough functional tests download a > > > > lot of assets from the internet, so if someone just runs "meson test" > > > > without further parameters, I think we should not trigger these downloads in > > > > that case. > > > > > > > > > > $ cat tests/Makefile.include > > > ... > > > check-functional: > > > @$(NINJA) precache-functional > > > @QEMU_TEST_NO_DOWNLOAD=1 $(MAKE) SPEED=thorough check-func > > > ... > > > > > > What's the rationale to run check-func with SPEED=thorough and > > > QEMU_TEST_NO_DOWNLOAD=1 with precache-functional having cached only quick > > > tests by default? > > > > Are you sure about that ? The precache logic uses custom_target() in > > meson and so does not filters on "suites" used to define quick vs > > thorough tests. > > > > My bad, I missed the "foreach speed : ['quick', 'thorough']" wrapping all > this in tests/functional/meson.build. > > > IOW, precache should always be downloading all assets. > > > > Then I don't understand the previous argument from Thomas to not make > thorough the default: "The thorough functional tests download a > lot of assets from the internet, so if someone just runs "meson test" > without further parameters, I think we should not trigger these downloads in > that case". It's what precache-functional is doing. I guess even without the download overhead, the tests using real VMs are going to be slower overall as they're doing more work to bring up a full OS install ? I have not quantified that difference though, and whether it is really due to the type of images being used, or whether it is just a perception from the fact that we'd be simply running more tests overall. With regards, Daniel
On 06/05/2025 17.31, Pierrick Bouvier wrote: > On 5/6/25 2:33 AM, Daniel P. Berrangé wrote: >> On Mon, May 05, 2025 at 10:46:52AM -0700, Pierrick Bouvier wrote: >>> On 5/5/25 3:32 AM, Thomas Huth wrote: >>>> On 03/05/2025 22.18, Pierrick Bouvier wrote: >>>>> Allows all tests to be visible by default when using meson test >>>>> directly. >>>>> >>>>> This has no impact on make check-* commands, which use SPEED=quick by >>>>> default (see scripts/mtest2make.py). >>>>> >>>>> Signed-off-by: Pierrick Bouvier <pierrick.bouvier@linaro.org> >>>>> --- >>>>> meson.build | 5 +++-- >>>>> 1 file changed, 3 insertions(+), 2 deletions(-) >>>>> >>>>> diff --git a/meson.build b/meson.build >>>>> index f8bf6e7bb66..57ff3f722d8 100644 >>>>> --- a/meson.build >>>>> +++ b/meson.build >>>>> @@ -5,12 +5,13 @@ project('qemu', ['c'], meson_version: '>=1.5.0', >>>>> meson.add_devenv({ 'MESON_BUILD_ROOT' : meson.project_build_root() }) >>>>> -add_test_setup('quick', exclude_suites: ['slow', 'thorough'], >>>>> is_default: true, >>>>> +add_test_setup('quick', exclude_suites: ['slow', 'thorough'], >>>>> env: ['RUST_BACKTRACE=1']) >>>>> add_test_setup('slow', exclude_suites: ['thorough'], >>>>> env: ['G_TEST_SLOW=1', 'SPEED=slow', >>>>> 'RUST_BACKTRACE=1']) >>>>> add_test_setup('thorough', >>>>> - env: ['G_TEST_SLOW=1', 'SPEED=thorough', >>>>> 'RUST_BACKTRACE=1']) >>>>> + env: ['G_TEST_SLOW=1', 'SPEED=thorough', >>>>> 'RUST_BACKTRACE=1'], >>>>> + is_default: true) >>>> >>>> I'd rather not make it the default: The thorough functional tests >>>> download a >>>> lot of assets from the internet, so if someone just runs "meson test" >>>> without further parameters, I think we should not trigger these >>>> downloads in >>>> that case. >>>> >>> >>> $ cat tests/Makefile.include >>> ... >>> check-functional: >>> @$(NINJA) precache-functional >>> @QEMU_TEST_NO_DOWNLOAD=1 $(MAKE) SPEED=thorough check-func >>> ... >>> >>> What's the rationale to run check-func with SPEED=thorough and >>> QEMU_TEST_NO_DOWNLOAD=1 with precache-functional having cached only quick >>> tests by default? >> >> Are you sure about that ? The precache logic uses custom_target() in >> meson and so does not filters on "suites" used to define quick vs >> thorough tests. >> > > My bad, I missed the "foreach speed : ['quick', 'thorough']" wrapping all > this in tests/functional/meson.build. > >> IOW, precache should always be downloading all assets. >> > > Then I don't understand the previous argument from Thomas to not make > thorough the default: "The thorough functional tests download a > lot of assets from the internet, so if someone just runs "meson test" > without further parameters, I think we should not trigger these downloads in > that case". It's what precache-functional is doing. precache-functional is *only* called when you run "make check-functional", i.e. when you know that you want to run the functional tests that might download assets from the internet. It's not called when you run the normal "make check". I think "meson test" should by default also not download any assets from the internet (e.g. in case someone is still on a metered internet connection or tries to do air-gapped builds) - we should only download if people really really want to run the functional tests consciously. Thomas
diff --git a/meson.build b/meson.build index f8bf6e7bb66..57ff3f722d8 100644 --- a/meson.build +++ b/meson.build @@ -5,12 +5,13 @@ project('qemu', ['c'], meson_version: '>=1.5.0', meson.add_devenv({ 'MESON_BUILD_ROOT' : meson.project_build_root() }) -add_test_setup('quick', exclude_suites: ['slow', 'thorough'], is_default: true, +add_test_setup('quick', exclude_suites: ['slow', 'thorough'], env: ['RUST_BACKTRACE=1']) add_test_setup('slow', exclude_suites: ['thorough'], env: ['G_TEST_SLOW=1', 'SPEED=slow', 'RUST_BACKTRACE=1']) add_test_setup('thorough', - env: ['G_TEST_SLOW=1', 'SPEED=thorough', 'RUST_BACKTRACE=1']) + env: ['G_TEST_SLOW=1', 'SPEED=thorough', 'RUST_BACKTRACE=1'], + is_default: true) meson.add_postconf_script(find_program('scripts/symlink-install-tree.py'))
Allows all tests to be visible by default when using meson test directly. This has no impact on make check-* commands, which use SPEED=quick by default (see scripts/mtest2make.py). Signed-off-by: Pierrick Bouvier <pierrick.bouvier@linaro.org> --- meson.build | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)