Do not retry failed tests if we reached the max number of failures

Context

!124444 (comment 1445099524)

What does this MR do and why?

Do not retry if we attained the max number of failures.

If we have RSPEC_FAIL_FAST_THRESHOLD tests failures, we might have more tests failures happening in the CI job.

If we only retry the RSPEC_FAIL_FAST_THRESHOLD tests that failed, we might miss other failures, and pass the CI job, which is not intended.

This MR ensures that we do not auto-retry the failed tests when we have the maximum of allowed failures.

Does it work?

Scenario 1 - Max failures reached (i.e. 20 failures)

• Test commit: !124583 (88d5e2a9)

Expectation

We should NOT auto-retry failed tests, and just fail the job.

Results

See https://gitlab.com/gitlab-org/gitlab/-/jobs/4558997784#L1226 and https://gitlab.com/gitlab-org/gitlab/-/jobs/4558960330#L1206:

Not retrying failing examples since we reached the maximum number of allowed test failures!

Scenario 2 - Max failures NOT reached (i.e. less than 20 failures)

• Test commit: !124583 (951d262c)

Expectation

We should auto-retry failed tests.

Results

See https://gitlab.com/gitlab-org/gitlab/-/jobs/4559152325#L1028 and https://gitlab.com/gitlab-org/gitlab/-/jobs/4559185042#L1078:

Finished in 2 minutes 5.7 seconds (files took 1 minute 5.06 seconds to load)
122 examples, 1 failure
Failed examples:
rspec ./spec/controllers/projects/pipelines_controller_spec.rb:39 # Projects::PipelinesController GET index.json when using persisted stages returns serialized pipelines
[TEST PROF INFO] Time spent in factories: 00:51.861 (33.99% of total time)
Stopped processing SimpleCov as a previous error not related to SimpleCov has been detected
RETRIED_TESTS_REPORT_PATH: rspec/flaky/retried_tests_rspec_fail-fast_report.txt
Retrying the failing examples in a new RSpec process...
$ gem install junit_merge --no-document --version 0.1.2
Successfully installed nokogiri-1.15.2-x86_64-linux
Successfully installed junit_merge-0.1.2
2 gems installed
==> 'gem install junit_merge --no-document --version 0.1.2' succeeded in 1 seconds.
Running RSpec command: bin/rspec -Ispec -rspec_helper --color --failure-exit-code 1 --error-exit-code 2 --format documentation --format Support::Formatters::JsonFormatter --out rspec/rspec-retry-4559152325.json --format RspecJunitFormatter --out rspec/rspec-retry-4559152325.xml --only-failures --pattern "{,ee/,jh/}spec/{,**/}*_spec.rb"

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

• I have evaluated the MR acceptance checklist for this MR.

scripts/rspec_helpers.sh

@@ -113,6 +113,8 @@ function rspec_simple_job() {
 }
 
 function rspec_simple_job_with_retry () {
+  set -o pipefail
+
   local rspec_run_status=0
 
   rspec_simple_job "${1}" "${2}" "${3}" | tee "${RSPEC_JOB_OUTPUT_PATH}"
@@ -151,7+153,7 @@
 
 function handle_retry_rspec_in_new_process() {
   local rspec_run_status="${1}"
 
   if [[ $rspec_run_status -eq 2 ]]; then
     echoerr "Not retrying failing examples since there were errors happening outside of the RSpec examples!"
   elif [[ $rspec_run_status -eq 1 ]]; then
@@ -177,7+179,7 @@
 }
 
 function rspec_paralellized_job() {
+  set -o pipefail
+
   read -ra job_name <<< "${CI_JOB_NAME}"
   local test_tool="${job_name[0]}"
   local test_level="${job_name[1]}"