| Message ID | 20221020115209.1761864-21-alex.bennee@linaro.org |
|---|---|
| State | New |
| Headers | show |
| Series | testing/next (docker, avocado, s390x, MAINTAINERS) | expand |
On 20/10/22 13:52, Alex Bennée wrote: > Running repeated invocations on a number of test boxes show a fairly > high error rate: > > $ retry.py -n 100 -c -- ./tests/venv/bin/avocado run tests/avocado/boot_linux_console.py:BootLinuxConsole.test_sh4_r2d > retry.py called with ['./tests/venv/bin/avocado', 'run', 'tests/avocado/boot_linux_console.py:BootLinuxConsole.test_sh4_r2d'] > Results: > Run, Ret, Pass/Fail, Time, Total Pass, Total Run > ... > Results summary: > 0: 94 times (94.00%), avg time 2.254 (0.00 varience/0.04 deviation) > 1: 3 times (3.00%), avg time 1.837 (0.02 varience/0.14 deviation) > 8: 3 times (3.00%), avg time 91.288 (0.02 varience/0.15 deviation) > > Examining the logs they fall into various categories of un-handled > unaligned access by user space and unexpected FPU usage by the kernel > which ultimately lead to the failure to reach the login prompt. This > could be bugs in the translator that only get hit occasionally or just > a flaky kernel - its hard to tell. To avoid these failures gating CI > lets skip on GitLab. > > Signed-off-by: Alex Bennée <alex.bennee@linaro.org> > Cc: Thomas Huth <thuth@redhat.com> > Cc: Yoshinori Sato <ysato@users.sourceforge.jp> > Cc: Magnus Damm <magnus.damm@gmail.com> > --- > tests/avocado/boot_linux_console.py | 4 ++++ > 1 file changed, 4 insertions(+) Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Alex Bennée <alex.bennee@linaro.org> writes: > Running repeated invocations on a number of test boxes show a fairly > high error rate: > > $ retry.py -n 100 -c -- ./tests/venv/bin/avocado run tests/avocado/boot_linux_console.py:BootLinuxConsole.test_sh4_r2d > retry.py called with ['./tests/venv/bin/avocado', 'run', 'tests/avocado/boot_linux_console.py:BootLinuxConsole.test_sh4_r2d'] > Results: > Run, Ret, Pass/Fail, Time, Total Pass, Total Run > ... > Results summary: > 0: 94 times (94.00%), avg time 2.254 (0.00 varience/0.04 deviation) > 1: 3 times (3.00%), avg time 1.837 (0.02 varience/0.14 deviation) > 8: 3 times (3.00%), avg time 91.288 (0.02 varience/0.15 deviation) > > Examining the logs they fall into various categories of un-handled > unaligned access by user space and unexpected FPU usage by the kernel > which ultimately lead to the failure to reach the login prompt. This > could be bugs in the translator that only get hit occasionally or just > a flaky kernel - its hard to tell. To avoid these failures gating CI > lets skip on GitLab. I've managed to bisect this regression to ab419fd8a0 (target/sh4: Fix TB_FLAG_UNALIGN). > > Signed-off-by: Alex Bennée <alex.bennee@linaro.org> > Cc: Thomas Huth <thuth@redhat.com> > Cc: Yoshinori Sato <ysato@users.sourceforge.jp> > Cc: Magnus Damm <magnus.damm@gmail.com> > --- > tests/avocado/boot_linux_console.py | 4 ++++ > 1 file changed, 4 insertions(+) > > diff --git a/tests/avocado/boot_linux_console.py b/tests/avocado/boot_linux_console.py > index eed4b49e6e..26eaf1a119 100644 > --- a/tests/avocado/boot_linux_console.py > +++ b/tests/avocado/boot_linux_console.py > @@ -1175,6 +1175,10 @@ def test_ppc_mac99(self): > self.vm.add_args('-M', 'graphics=off') > self.do_test_advcal_2018('15', tar_hash, 'invaders.elf') > > + # This test has a 6-10% failure rate on various hosts that look > + # like issues with a buggy kernel. As a result we don't want it > + # gating releases on Gitlab. > + @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab') > def test_sh4_r2d(self): > """ > :avocado: tags=arch:sh4
Alex Bennée <alex.bennee@linaro.org> writes: > Running repeated invocations on a number of test boxes show a fairly > high error rate: > > $ retry.py -n 100 -c -- ./tests/venv/bin/avocado run tests/avocado/boot_linux_console.py:BootLinuxConsole.test_sh4_r2d > retry.py called with ['./tests/venv/bin/avocado', 'run', 'tests/avocado/boot_linux_console.py:BootLinuxConsole.test_sh4_r2d'] > Results: > Run, Ret, Pass/Fail, Time, Total Pass, Total Run > ... > Results summary: > 0: 94 times (94.00%), avg time 2.254 (0.00 varience/0.04 deviation) > 1: 3 times (3.00%), avg time 1.837 (0.02 varience/0.14 deviation) > 8: 3 times (3.00%), avg time 91.288 (0.02 varience/0.15 deviation) > > Examining the logs they fall into various categories of un-handled > unaligned access by user space and unexpected FPU usage by the kernel > which ultimately lead to the failure to reach the login prompt. This > could be bugs in the translator that only get hit occasionally or just > a flaky kernel - its hard to tell. To avoid these failures gating CI > lets skip on GitLab. > > Signed-off-by: Alex Bennée <alex.bennee@linaro.org> > Cc: Thomas Huth <thuth@redhat.com> > Cc: Yoshinori Sato <ysato@users.sourceforge.jp> > Cc: Magnus Damm <magnus.damm@gmail.com> > --- > tests/avocado/boot_linux_console.py | 4 ++++ > 1 file changed, 4 insertions(+) > > diff --git a/tests/avocado/boot_linux_console.py b/tests/avocado/boot_linux_console.py > index eed4b49e6e..26eaf1a119 100644 > --- a/tests/avocado/boot_linux_console.py > +++ b/tests/avocado/boot_linux_console.py > @@ -1175,6 +1175,10 @@ def test_ppc_mac99(self): > self.vm.add_args('-M', 'graphics=off') > self.do_test_advcal_2018('15', tar_hash, 'invaders.elf') > > + # This test has a 6-10% failure rate on various hosts that look > + # like issues with a buggy kernel. As a result we don't want it > + # gating releases on Gitlab. > + @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab') I also forgot to add the import: from avocado import skipIf > def test_sh4_r2d(self): > """ > :avocado: tags=arch:sh4
diff --git a/tests/avocado/boot_linux_console.py b/tests/avocado/boot_linux_console.py index eed4b49e6e..26eaf1a119 100644 --- a/tests/avocado/boot_linux_console.py +++ b/tests/avocado/boot_linux_console.py @@ -1175,6 +1175,10 @@ def test_ppc_mac99(self): self.vm.add_args('-M', 'graphics=off') self.do_test_advcal_2018('15', tar_hash, 'invaders.elf') + # This test has a 6-10% failure rate on various hosts that look + # like issues with a buggy kernel. As a result we don't want it + # gating releases on Gitlab. + @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab') def test_sh4_r2d(self): """ :avocado: tags=arch:sh4
Running repeated invocations on a number of test boxes show a fairly high error rate: $ retry.py -n 100 -c -- ./tests/venv/bin/avocado run tests/avocado/boot_linux_console.py:BootLinuxConsole.test_sh4_r2d retry.py called with ['./tests/venv/bin/avocado', 'run', 'tests/avocado/boot_linux_console.py:BootLinuxConsole.test_sh4_r2d'] Results: Run, Ret, Pass/Fail, Time, Total Pass, Total Run ... Results summary: 0: 94 times (94.00%), avg time 2.254 (0.00 varience/0.04 deviation) 1: 3 times (3.00%), avg time 1.837 (0.02 varience/0.14 deviation) 8: 3 times (3.00%), avg time 91.288 (0.02 varience/0.15 deviation) Examining the logs they fall into various categories of un-handled unaligned access by user space and unexpected FPU usage by the kernel which ultimately lead to the failure to reach the login prompt. This could be bugs in the translator that only get hit occasionally or just a flaky kernel - its hard to tell. To avoid these failures gating CI lets skip on GitLab. Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Cc: Thomas Huth <thuth@redhat.com> Cc: Yoshinori Sato <ysato@users.sourceforge.jp> Cc: Magnus Damm <magnus.damm@gmail.com> --- tests/avocado/boot_linux_console.py | 4 ++++ 1 file changed, 4 insertions(+)