Do not retry failed tests if we reached the max number of failures
Context
What does this MR do and why?
Do not retry if we attained the max number of failures.
If we have RSPEC_FAIL_FAST_THRESHOLD tests failures, we might
have more tests failures happening in the CI job.
If we only retry the RSPEC_FAIL_FAST_THRESHOLD tests that failed,
we might miss other failures, and pass the CI job,
which is not intended.
This MR ensures that we do not auto-retry the failed tests when we have the maximum of allowed failures.
Does it work?
Scenario 1 - Max failures reached (i.e. 20 failures)
- Test commit: !124583 (88d5e2a9)
Expectation
We should NOT auto-retry failed tests, and just fail the job.
Results
See https://gitlab.com/gitlab-org/gitlab/-/jobs/4558997784#L1226 and https://gitlab.com/gitlab-org/gitlab/-/jobs/4558960330#L1206:
Not retrying failing examples since we reached the maximum number of allowed test failures!Scenario 2 - Max failures NOT reached (i.e. less than 20 failures)
- Test commit: !124583 (951d262c)
Expectation
We should auto-retry failed tests.
Results
See https://gitlab.com/gitlab-org/gitlab/-/jobs/4559152325#L1028 and https://gitlab.com/gitlab-org/gitlab/-/jobs/4559185042#L1078:
Finished in 2 minutes 5.7 seconds (files took 1 minute 5.06 seconds to load)
122 examples, 1 failure
Failed examples:
rspec ./spec/controllers/projects/pipelines_controller_spec.rb:39 # Projects::PipelinesController GET index.json when using persisted stages returns serialized pipelines
[TEST PROF INFO] Time spent in factories: 00:51.861 (33.99% of total time)
Stopped processing SimpleCov as a previous error not related to SimpleCov has been detected
RETRIED_TESTS_REPORT_PATH: rspec/flaky/retried_tests_rspec_fail-fast_report.txt
Retrying the failing examples in a new RSpec process...
$ gem install junit_merge --no-document --version 0.1.2
Successfully installed nokogiri-1.15.2-x86_64-linux
Successfully installed junit_merge-0.1.2
2 gems installed
==> 'gem install junit_merge --no-document --version 0.1.2' succeeded in 1 seconds.
Running RSpec command: bin/rspec -Ispec -rspec_helper --color --failure-exit-code 1 --error-exit-code 2 --format documentation --format Support::Formatters::JsonFormatter --out rspec/rspec-retry-4559152325.json --format RspecJunitFormatter --out rspec/rspec-retry-4559152325.xml --only-failures --pattern "{,ee/,jh/}spec/{,**/}*_spec.rb"MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.