Skip to content

Run all backend specs in random order

Problem

In !93137 (merged) we started running all 🆕 backend spec in random order.

To maintainer master stability, all existing backend specs were excluded (via spec/support/rspec_order_todo.yml) from running in random order so they still run in "defined" order (rspec --order defined).

Proposed solution

To make spec files run in random order, check their order dependency with:

scripts/rspec_check_order_dependence <path> ...
scripts/rspec_check_order_dependence spec/models/project_spec.rb
scripts/rspec_check_order_dependence spec/models/project_spec.rb spec/models/project_label_spec.rb
scripts/rspec_check_order_dependence spec/models/project*

If the specs pass the check the script removes them from rspec_order_todo.yml automatically.

If the specs fail the check they must be fixed before they can run in random order.

See https://docs.gitlab.com/ee/development/testing_guide/best_practices.html#test-order

The list

As of 2023-04-19 the list spec/support/rspec_order_todo.yml contains over 10.000 entries.

Implementation Guide

Any help is much appreciated ❤

  1. Pick entries from spec/support/rspec_order_todo.yml
  2. Check their order dependency via scripts/rspec_check_order_dependence
  3. If specs
  • ✅ Pass, commit the changes in spec/support/rspec_order_todo.yml
  • ❌ Fail
    • 🅰 Fix the specs and go to step 2
    • 🅱 Create an issue and ping @splattael for triage
  1. Submit an MR and mention this issue via Contributes to #407877
  2. 🎉

Example MRs

Edited by Peter Leitzen