Skip to content

ci: Retry failed specs in a new process after the initial run

Rémy Coutable requested to merge retry-failing-tests-2 into master

What does this MR do and why?

This MR add automatic retries for failed tests in a new bin/rspec process. The goal is to get rid of most side-effects from previous tests that may lead to a subsequent test failure.

See https://gitlab.com/gitlab-org/gitlab/-/jobs/1904232583#L8418 as an example.

This also takes care of merging the JUnit report from the retry into the main run report, so that if the tests pass on retry, the report will be completely successful.

The list of tests that are retried in a new RSpec process is computed in an artifact by the rspec:flaky-tests-report job, e.g. https://gitlab.com/gitlab-org/gitlab/-/jobs/1998015640/artifacts/file/rspec/flaky/retried_tests_report.txt

Experiment issue: https://gitlab.com/gitlab-org/quality/team-tasks/-/issues/1148.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Rémy Coutable

Merge request reports