Run all new backend specs in random order

What does this MR do and why?

With this MR all 🆕 specs are run in random order to surface flaky tests that are dependent on test order.

All (minus spec/rubocop - see !93143 (merged)) old specs are still run in defined order and excluded via spec/support/rspec_order_todo.yml (10930 files).

Note that the example group description ends with " (random)" if run in random order to help debugging spec failures. See example below.

This MR also enables all RuboCop specs to be run in random order. This has been checked via scripts/rspec_check_order_dependence spec/rubocop which checks passed spec files and deletes them from spec/support/rspec_order_todo.yml on success 🎉. See screenshot below.

Refs #337399 (closed).

Mixed mode

Note that running specs in random and defined order still works just fine because randomization happens on example group basis.

For example:

bin/rspec spec/lib/gitlab_spec.rb spec/rubocop/cop_todo_spec.rb
Gitlab
  delegates root to GitlabEdition
  delegates jh? to GitlabEdition
  delegates ee? to GitlabEdition
  delegates extensions to GitlabEdition
...

RuboCop::CopTodo (random)
  #to_yaml
    when previously disabled
...

Gitlab example group is run in defined order where as RuboCop::CopTodo (random) (note the (random) appendix is run in random order.

Reverse order?

When passing RSPEC_ORDER=reverse you can run specs in reverse order which is used by scripts/rspec_check_order_dependence.

Reproducibility

In order to reproduce spec failures due to order issues you can use the seed via --seed SEED. Example: Randomized with seed 27443.

qa/?

QA specs are already run in random order and are out of scope of this MR.

Screenshots or screen recordings

scripts/rspec_check_order_dependence spec/rubocop - see !93143 (merged)

How to set up and validate locally

bin/rspec --format documentation spec/rubocop

Run options: include {:focus=>true}

All examples were filtered out; ignoring {:focus=>true}

Randomized with seed 22503

RuboCop::Cop::Migration::Datetime (random)
  when in migration
    registers an offense when the ":datetime" data type is used on add_column
    registers an offense when the ":timestamp" data type is used on create_table
    does not register an offense when the ":datetime_with_timezone" data type is used on create_table
    does not register an offense when the ":datetime" data type is not used on add_column
    does not register an offense when the ":datetime_with_timezone" data type is used on add_column
    registers an offense when the ":timestamp" data type is used on add_column
    does not register an offense when the ":datetime" data type is not used on create_table
    registers an offense when the ":datetime" data type is used on create_table
  when outside of migration
    registers no offense

RuboCop::Cop::AvoidReturnFromBlocks (random)
  doesn't create more than one offense for nested blocks
  doesn't flag violation for return used inside a method definition
  doesn't flag violation for next inside a block
  doesn't check when block is empty
  doesn't flag violation for return inside a lambda
  flags violation for return inside a block
  flags violation for return inside included > def > block
  doesn't flag violation for break inside a block
  behaves like examples with def methods
    doesn't flag violation for return inside define_method
  behaves like examples with def methods
    doesn't flag violation for return inside lambda
  behaves like examples with whitelisted method
    doesn't flag violation for return inside loop
  behaves like examples with whitelisted method
    doesn't flag violation for return inside times
  behaves like examples with whitelisted method
    doesn't flag violation for return inside each
  behaves like examples with whitelisted method
    doesn't flag violation for return inside each_filename
...

Note the (random) appendix to each example group to denote that it's run in random order.

TODO YAML

Generated via

find spec ee/spec -type f -name "*.rb" | grep -E "_spec\.rb$|frontend/fixtures" | sort | perl -lne "print qq{- './\$_'}" > spec/support/rspec_order_todo.yml

Current TODOs are:

• All spec files ending with _spec.rb
• All frontend fixtures (matching frontend/fixtures)

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

• I have evaluated the MR acceptance checklist for this MR.

doc/development/testing_guide/best_practices.md

@@ -87,22 +87,21 @@ SILENCE_DEPRECATIONS=1 bin/rspec spec/models/project_spec.rb
 
 > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/93137) in GitLab 15.3.
 
-All _new_ specs are run in [random order](https://gitlab.com/gitlab-org/gitlab/-/issues/337399)
+All _new_ spec files are run in [random order](https://gitlab.com/gitlab-org/gitlab/-/issues/337399)
 to surface flaky tests that are dependent test order.
 
 For a list of spec files which are still run in defined order, see [`rspec_order_todo.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/spec/support/rspec_order_todo.yml).
 
-To make specs run in random order:
+To make spec files run in random order, check their order dependency with:
 
-1. Check if specs fail when run in random order:
-
-    ```shell
-    scripts/rspec_check_order_dependence spec/models/project_spec.rb
-    ```
+```shell
+scripts/rspec_check_order_dependence spec/models/project_spec.rb
+```
 
-1. Fix the spec failures if needed
+If they pass the check the spec files the script removes them from
+[`rspec_order_todo.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/spec/support/rspec_order_todo.yml).
 
-1. Remove checked spec files from [`rspec_order_todo.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/spec/support/rspec_order_todo.yml).
+If they fail the check spec failures must be fixed.
 
 ### Test speed