Retry rspec in a new process should retry the file when there's an error
Automatic retry of failing tests in a separate process: https://docs.gitlab.com/ee/development/pipelines/index.html#automatic-retry-of-failing-tests-in-a-separate-process
In rspec unit pg12 minimal 4/28 we noticed that if there's an error in the test:
An error occurred in an `after(:context)` hook.
Failure/Error: MarginaliaTestJob.clear
NameError:
uninitialized constant MarginaliaTestJob
# ./spec/lib/marginalia_spec.rb:101:in `block (3 levels) in <top (required)>'
Finished in 4.58 seconds (files took 1 minute 1.3 seconds to load)
5 examples, 0 failures, 1 error occurred outside of examples
The follow up retry RSpec process did not retry it:
Test environment set up in 1.876373895 seconds
Finished in 4.37 seconds (files took 2 minutes 34.1 seconds to load)
0 examples, 0 failures
This caused the job to be green, when it should be red.
The same error was detected in rspec fail-fast because it did not have this mechanism.
This was found in https://gitlab.com/gitlab-org/quality/triage-ops/-/issues/1165#note_1209180703
Edited by Lin Jen-Shin