Skip to content

Run specs in random order multiple times

Peter Leitzen requested to merge pl-rspec-check-order-deps-runs into master

What does this MR do and why?

Sometimes, running specs in random order only once is not sufficient to catch flaky specs.

This MR also fixes running specs in random order with spring when enabled.

Refs !157178 (comment 1968533439)

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

How to set up and validate locally

scripts/rspec_check_order_dependence ./ee/spec/config/metrics/every_metric_definition_spec.rb
# See spec being run 7 times:
# - 1x defined order
# - 1x reverse order
# - 5x random order

Running in defined order:
Running in reverse order:
Running in random order (1/5):
Running in random order (2/5):
Running in random order (3/5):
Running in random order (4/5):
Running in random order (5/5):


export ENABLE_SPRING=1
bin/rspec ./ee/spec/config/metrics/every_metric_definition_spec.rb

bin/rspec ./ee/spec/config/metrics/every_metric_definition_spec.rb
# Seed is now different

bin/rspec --seed 42 ./ee/spec/config/metrics/every_metric_definition_spec.rb
# Seed is now pinned
Edited by Peter Leitzen

Merge request reports