Do not retry failed tests if we reached the max number of failures

Context

!124444 (comment 1445099524)

What does this MR do and why?

Do not retry if we attained the max number of failures.

If we have RSPEC_FAIL_FAST_THRESHOLD tests failures, we might have more tests failures happening in the CI job.

If we only retry the RSPEC_FAIL_FAST_THRESHOLD tests that failed, we might miss other failures, and pass the CI job, which is not intended.

This MR ensures that we do not auto-retry the failed tests when we have the maximum of allowed failures.

Does it work?

Scenario 1 - Max failures reached (i.e. 20 failures)

• Test commit: !124583 (88d5e2a9)

Expectation

We should NOT auto-retry failed tests, and just fail the job.

Results

See https://gitlab.com/gitlab-org/gitlab/-/jobs/4558997784#L1226 and https://gitlab.com/gitlab-org/gitlab/-/jobs/4558960330#L1206:

Not retrying failing examples since we reached the maximum number of allowed test failures!

Scenario 2 - Max failures NOT reached (i.e. less than 20 failures)

• Test commit: !124583 (951d262c)

Expectation

We should auto-retry failed tests.

Results

See https://gitlab.com/gitlab-org/gitlab/-/jobs/4559152325#L1028 and https://gitlab.com/gitlab-org/gitlab/-/jobs/4559185042#L1078:

Finished in 2 minutes 5.7 seconds (files took 1 minute 5.06 seconds to load)
122 examples, 1 failure
Failed examples:
rspec ./spec/controllers/projects/pipelines_controller_spec.rb:39 # Projects::PipelinesController GET index.json when using persisted stages returns serialized pipelines
[TEST PROF INFO] Time spent in factories: 00:51.861 (33.99% of total time)
Stopped processing SimpleCov as a previous error not related to SimpleCov has been detected
RETRIED_TESTS_REPORT_PATH: rspec/flaky/retried_tests_rspec_fail-fast_report.txt
Retrying the failing examples in a new RSpec process...
$ gem install junit_merge --no-document --version 0.1.2
Successfully installed nokogiri-1.15.2-x86_64-linux
Successfully installed junit_merge-0.1.2
2 gems installed
==> 'gem install junit_merge --no-document --version 0.1.2' succeeded in 1 seconds.
Running RSpec command: bin/rspec -Ispec -rspec_helper --color --failure-exit-code 1 --error-exit-code 2 --format documentation --format Support::Formatters::JsonFormatter --out rspec/rspec-retry-4559152325.json --format RspecJunitFormatter --out rspec/rspec-retry-4559152325.xml --only-failures --pattern "{,ee/,jh/}spec/{,**/}*_spec.rb"

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

• I have evaluated the MR acceptance checklist for this MR.

assigned to @ddieulivol

added backend label

marked the checklist item I have evaluated the MR acceptance checklist for this MR. as completed

removed backend label

	1 Message
	CHANGELOG missing: If you want to create a changelog entry for GitLab FOSS, add the Changelog trailer to the commit message you want to add to the changelog. If you want to create a changelog entry for GitLab EE, also add the EE: true trailer to your commit message. If this merge request doesn't need a CHANGELOG entry, feel free to ignore this message.

Reviewer roulette

Changes that require review have been detected!

Please refer to the table below for assigning reviewers and maintainers suggested by Danger in the specified category:

Category	Reviewer	Maintainer
maintenanceworkflow / maintenancepipelines for CI, Danger	Eduardo Bonet (@eduardobonet) (UTC+2, same timezone as @ddieulivol)	Nicolas Dular (@nicolasdular) (UTC+2, same timezone as @ddieulivol)
Engineering Productivity	Reviewer review is optional for Engineering Productivity	Jennifer Li (@jennli) (UTC-7, 9 hours behind @ddieulivol)

To spread load more evenly across eligible reviewers, Danger has picked a candidate for each review slot, based on their timezone. Feel free to override these selections if you think someone else would be better-suited or use the GitLab Review Workload Dashboard to find other available reviewers.

To read more on how to use the reviewer roulette, please take a look at the Engineering workflow and code review guidelines. Please consider assigning a reviewer or maintainer who is a domain expert in the area of the merge request.

Once you've decided who will review this merge request, assign them as a reviewer! Danger does not automatically notify them for you.

If needed, you can retry the danger-review job that generated this comment.

Generated by Danger

mentioned in merge request !124444 (merged)

Proper labels assigned to this merge request. Please ignore me.

@ddieulivol - please see the following guidance and update this merge request.

	1 Error
	Please add typebug typefeature, or typemaintenance label to this merge request.

changed the description

added 2 commits

• cc4d63c3 - Do not retry if we have the max number of failures
• 19394abd - TEST - Fail specs on purpose

Compare with previous version

added backend label

changed milestone to %16.2

added Engineering Productivity maintenanceperformance + 1 deleted label

added typemaintenance label

added 1 commit

• 2787b2f0 - Experiment with pipefail

Compare with previous version

resolved all threads

added 38 commits

• 2787b2f0...59409bf0 - 36 commits from branch master
• e4424939 - Do not retry if we have the max number of failures
• 8787f4b6 - TEST - Fail specs on purpose

Compare with previous version

added 1 commit

• ef1b32a3 - Add shell options debugging

Compare with previous version

added 1 commit

• 0186338e - Inline the status

Compare with previous version

changed the description