diff mbox series

[v3,24/24] tests/avocado: disable BootLinuxPPC64 test in CI

Message ID 20230228190653.1602033-25-alex.bennee@linaro.org
State Superseded
Headers show
Series testing/next: docker, avocado, unit, gitlab | expand

Commit Message

Alex Bennée Feb. 28, 2023, 7:06 p.m. UTC
This test is exceptionally heavyweight (nearly 330s) compared to the
two (both endians) TuxRun baseline tests which complete in under 160s.
The coverage is slightly reduced but a more directed test could make
up the difference.

tests/avocado/tuxrun_baselines.py:TuxRunBaselineTest.test_ppc64:

Overall coverage rate:
  lines......: 9.6% (44110 of 458817 lines)
  functions..: 16.5% (6767 of 41054 functions)
  branches...: 6.0% (13395 of 222634 branches)

tests/avocado/boot_linux.py:BootLinuxPPC64.test_pseries_tcg:

Overall coverage rate:
  lines......: 11.6% (53408 of 458817 lines)
  functions..: 18.7% (7691 of 41054 functions)
  branches...: 7.9% (17692 of 224218 branches)

So lets skip for GITLAB_CI and also unless AVOCADO_TIMEOUT_EXPECTED is
specified by the user.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
 tests/avocado/boot_linux.py | 4 ++++
 1 file changed, 4 insertions(+)

Comments

Philippe Mathieu-Daudé Feb. 28, 2023, 9:53 p.m. UTC | #1
On 28/2/23 20:06, Alex Bennée wrote:
> This test is exceptionally heavyweight (nearly 330s) compared to the
> two (both endians) TuxRun baseline tests which complete in under 160s.
> The coverage is slightly reduced but a more directed test could make
> up the difference.
> 
> tests/avocado/tuxrun_baselines.py:TuxRunBaselineTest.test_ppc64:
> 
> Overall coverage rate:
>    lines......: 9.6% (44110 of 458817 lines)
>    functions..: 16.5% (6767 of 41054 functions)
>    branches...: 6.0% (13395 of 222634 branches)
> 
> tests/avocado/boot_linux.py:BootLinuxPPC64.test_pseries_tcg:
> 
> Overall coverage rate:
>    lines......: 11.6% (53408 of 458817 lines)
>    functions..: 18.7% (7691 of 41054 functions)
>    branches...: 7.9% (17692 of 224218 branches)
> 
> So lets skip for GITLAB_CI and also unless AVOCADO_TIMEOUT_EXPECTED is
> specified by the user.
> 
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> ---
>   tests/avocado/boot_linux.py | 4 ++++
>   1 file changed, 4 insertions(+)

Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Thomas Huth March 1, 2023, 7:14 a.m. UTC | #2
On 28/02/2023 20.06, Alex Bennée wrote:
> This test is exceptionally heavyweight (nearly 330s) compared to the
> two (both endians) TuxRun baseline tests which complete in under 160s.
> The coverage is slightly reduced but a more directed test could make
> up the difference.
> 
> tests/avocado/tuxrun_baselines.py:TuxRunBaselineTest.test_ppc64:
> 
> Overall coverage rate:
>    lines......: 9.6% (44110 of 458817 lines)
>    functions..: 16.5% (6767 of 41054 functions)
>    branches...: 6.0% (13395 of 222634 branches)
> 
> tests/avocado/boot_linux.py:BootLinuxPPC64.test_pseries_tcg:
> 
> Overall coverage rate:
>    lines......: 11.6% (53408 of 458817 lines)
>    functions..: 18.7% (7691 of 41054 functions)
>    branches...: 7.9% (17692 of 224218 branches)
> 
> So lets skip for GITLAB_CI and also unless AVOCADO_TIMEOUT_EXPECTED is
> specified by the user.

The explanation sounds somewhat implausible to me. AVOCADO_TIMEOUT_EXPECTED 
should be for jobs where we are not sure whether the job really finishes in 
time, e.g. when compiling QEMU with debug flags enabled, and not for jobs 
that simply run a little bit longer (in the latter case, it would be enough 
to simply bump the timeout setting a little bit if necessary). So did you 
check whether you really run into timeout issues here when compiling QEMU 
with debug flags?

Anyway, if you add AVOCADO_TIMEOUT_EXPECTED, then I think you don't need 
GITLAB_CI anymore, since we certainly don't set AVOCADO_TIMEOUT_EXPECTED in 
the gitlab CI.

  Thomas
Alex Bennée March 1, 2023, 9:57 a.m. UTC | #3
Thomas Huth <thuth@redhat.com> writes:

> On 28/02/2023 20.06, Alex Bennée wrote:
>> This test is exceptionally heavyweight (nearly 330s) compared to the
>> two (both endians) TuxRun baseline tests which complete in under 160s.
>> The coverage is slightly reduced but a more directed test could make
>> up the difference.
>> tests/avocado/tuxrun_baselines.py:TuxRunBaselineTest.test_ppc64:
>> Overall coverage rate:
>>    lines......: 9.6% (44110 of 458817 lines)
>>    functions..: 16.5% (6767 of 41054 functions)
>>    branches...: 6.0% (13395 of 222634 branches)
>> tests/avocado/boot_linux.py:BootLinuxPPC64.test_pseries_tcg:
>> Overall coverage rate:
>>    lines......: 11.6% (53408 of 458817 lines)
>>    functions..: 18.7% (7691 of 41054 functions)
>>    branches...: 7.9% (17692 of 224218 branches)
>> So lets skip for GITLAB_CI and also unless AVOCADO_TIMEOUT_EXPECTED
>> is
>> specified by the user.
>
> The explanation sounds somewhat implausible to me.
> AVOCADO_TIMEOUT_EXPECTED should be for jobs where we are not sure
> whether the job really finishes in time, e.g. when compiling QEMU with
> debug flags enabled, and not for jobs that simply run a little bit
> longer (in the latter case, it would be enough to simply bump the
> timeout setting a little bit if necessary). So did you check whether
> you really run into timeout issues here when compiling QEMU with debug
> flags?

Ahh I realise now that I was running into the timeout because it was a
gcov build. I'll drop the AVOACADO_TIMEOUT_EXPECTED bit for now.

>
> Anyway, if you add AVOCADO_TIMEOUT_EXPECTED, then I think you don't
> need GITLAB_CI anymore, since we certainly don't set
> AVOCADO_TIMEOUT_EXPECTED in the gitlab CI.
>
>  Thomas
Philippe Mathieu-Daudé March 1, 2023, 11:12 a.m. UTC | #4
On 1/3/23 10:57, Alex Bennée wrote:
> 
> Thomas Huth <thuth@redhat.com> writes:
> 
>> On 28/02/2023 20.06, Alex Bennée wrote:
>>> This test is exceptionally heavyweight (nearly 330s) compared to the
>>> two (both endians) TuxRun baseline tests which complete in under 160s.
>>> The coverage is slightly reduced but a more directed test could make
>>> up the difference.
>>> tests/avocado/tuxrun_baselines.py:TuxRunBaselineTest.test_ppc64:
>>> Overall coverage rate:
>>>     lines......: 9.6% (44110 of 458817 lines)
>>>     functions..: 16.5% (6767 of 41054 functions)
>>>     branches...: 6.0% (13395 of 222634 branches)
>>> tests/avocado/boot_linux.py:BootLinuxPPC64.test_pseries_tcg:
>>> Overall coverage rate:
>>>     lines......: 11.6% (53408 of 458817 lines)
>>>     functions..: 18.7% (7691 of 41054 functions)
>>>     branches...: 7.9% (17692 of 224218 branches)
>>> So lets skip for GITLAB_CI and also unless AVOCADO_TIMEOUT_EXPECTED
>>> is
>>> specified by the user.
>>
>> The explanation sounds somewhat implausible to me.
>> AVOCADO_TIMEOUT_EXPECTED should be for jobs where we are not sure
>> whether the job really finishes in time, e.g. when compiling QEMU with
>> debug flags enabled, and not for jobs that simply run a little bit
>> longer (in the latter case, it would be enough to simply bump the
>> timeout setting a little bit if necessary). So did you check whether
>> you really run into timeout issues here when compiling QEMU with debug
>> flags?
> 
> Ahh I realise now that I was running into the timeout because it was a
> gcov build. I'll drop the AVOACADO_TIMEOUT_EXPECTED bit for now.
> 
>>
>> Anyway, if you add AVOCADO_TIMEOUT_EXPECTED, then I think you don't
>> need GITLAB_CI anymore, since we certainly don't set
>> AVOCADO_TIMEOUT_EXPECTED in the gitlab CI.

Correct we won't run AVOCADO_TIMEOUT_EXPECTED on GitLab CI.
Note however GITLAB_CI and AVOCADO_TIMEOUT_EXPECTED are not orthogonal.
See docs/devel/testing.rst:

AVOCADO_TIMEOUT_EXPECTED
...
   Even though the timeout can be set by the test developer, there
   are some tests that may not have a well-defined limit of time
   to finish under certain conditions. For example, tests that take
   longer to execute when QEMU is compiled with debug flags.
   Therefore, the ``AVOCADO_TIMEOUT_EXPECTED`` variable has been
   used to determine whether those tests should run or not.

GITLAB_CI
   A number of tests are flagged to not run on the GitLab CI.
   Usually because they proved to the flaky or there are constraints
   on the CI environment which would make them fail.

Personally I run the Avocado tests on a daily basis without
AVOCADO_TIMEOUT_EXPECTED, but certainly set it before posting
a pull request.
diff mbox series

Patch

diff --git a/tests/avocado/boot_linux.py b/tests/avocado/boot_linux.py
index fe0bb180d9..f09f7b1dbf 100644
--- a/tests/avocado/boot_linux.py
+++ b/tests/avocado/boot_linux.py
@@ -84,6 +84,8 @@  def test_virt_kvm(self):
         self.launch_and_wait(set_up_ssh_connection=False)
 
 
+# See the tux_baseline.py tests for almost the same coverage in a lot
+# less time.
 class BootLinuxPPC64(LinuxTest):
     """
     :avocado: tags=arch:ppc64
@@ -91,6 +93,8 @@  class BootLinuxPPC64(LinuxTest):
 
     timeout = 360
 
+    @skipUnless(os.getenv('AVOCADO_TIMEOUT_EXPECTED'), 'Test might timeout')
+    @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab')
     def test_pseries_tcg(self):
         """
         :avocado: tags=machine:pseries