Skip to content

ci: Fail RSpec jobs when errors occur outside of examples

What does this MR do and why?

When an error happens outside an example (i.e. in a hook), we now fail the job directly, without retrying any failed examples.

The problem was that examples that error out are listed as passed in RSpec's report, so that they weren't retried and the job could pass as a result even though we actually had examples in an error state!

Related to #385752 (closed).

Screenshots or screen recordings

Example of a test that errors out: https://gitlab.com/gitlab-org/gitlab/-/jobs/3582179391

Pending: (Failures listed here are expected and do not affect your suite's status)
  1) Operations::FeatureFlag validations behaves like AtomicInternalId .has_internal_id Validation when presence validation is not required when creating an object does not raise an error if the internal id is blank
     # No reason given
     # ./spec/support/shared_examples/models/atomic_internal_id_shared_examples.rb:49
  2) Operations::FeatureFlag validations behaves like AtomicInternalId .has_internal_id Validation when presence validation is not required when updating an object does not raise an error if the internal id is blank
     # No reason given
     # ./spec/support/shared_examples/models/atomic_internal_id_shared_examples.rb:57
Finished in 27 minutes 24 seconds (files took 1 minute 14.15 seconds to load)
3371 examples, 0 failures, 2 pending, 1 error occurred outside of examples
Randomized with seed 28879
[TEST PROF INFO] Time spent in factories: 17:05.866 (61.21% of total time)
Stopped processing SimpleCov as a previous error not related to SimpleCov has been detected
RSpec exited with 1.
Not retrying failing examples since there were errors happening outside of the RSpec examples!

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Rémy Coutable

Merge request reports