Run all new backend specs in random order
What does this MR do and why?
With this MR all
All (minus spec/rubocop
- see !93143 (merged)) old specs are still run in defined order and excluded via
spec/support/rspec_order_todo.yml
(10930 files).
Note that the example group description ends with " (random)" if run in random order to help debugging spec failures. See example below.
This MR also enables all RuboCop specs to be run in random order. This has been checked via scripts/rspec_check_order_dependence spec/rubocop
which checks passed spec files and deletes them from spec/support/rspec_order_todo.yml
on success
Refs #337399 (closed).
Mixed mode
Note that running specs in random and defined order still works just fine because randomization happens on example group basis.
For example:
bin/rspec spec/lib/gitlab_spec.rb spec/rubocop/cop_todo_spec.rb
Gitlab
delegates root to GitlabEdition
delegates jh? to GitlabEdition
delegates ee? to GitlabEdition
delegates extensions to GitlabEdition
...
RuboCop::CopTodo (random)
#to_yaml
when previously disabled
...
Gitlab
example group is run in defined order where as RuboCop::CopTodo (random)
(note the (random)
appendix is run in random order.
Reverse order?
When passing RSPEC_ORDER=reverse
you can run specs in reverse order which is used by scripts/rspec_check_order_dependence
.
Reproducibility
In order to reproduce spec failures due to order issues you can use the seed via --seed SEED
. Example: Randomized with seed 27443
.
qa/
?
QA specs are already run in random order and are out of scope of this MR.
Screenshots or screen recordings
scripts/rspec_check_order_dependence spec/rubocop - see !93143 (merged)
|
---|
How to set up and validate locally
bin/rspec --format documentation spec/rubocop
Run options: include {:focus=>true}
All examples were filtered out; ignoring {:focus=>true}
Randomized with seed 22503
RuboCop::Cop::Migration::Datetime (random)
when in migration
registers an offense when the ":datetime" data type is used on add_column
registers an offense when the ":timestamp" data type is used on create_table
does not register an offense when the ":datetime_with_timezone" data type is used on create_table
does not register an offense when the ":datetime" data type is not used on add_column
does not register an offense when the ":datetime_with_timezone" data type is used on add_column
registers an offense when the ":timestamp" data type is used on add_column
does not register an offense when the ":datetime" data type is not used on create_table
registers an offense when the ":datetime" data type is used on create_table
when outside of migration
registers no offense
RuboCop::Cop::AvoidReturnFromBlocks (random)
doesn't create more than one offense for nested blocks
doesn't flag violation for return used inside a method definition
doesn't flag violation for next inside a block
doesn't check when block is empty
doesn't flag violation for return inside a lambda
flags violation for return inside a block
flags violation for return inside included > def > block
doesn't flag violation for break inside a block
behaves like examples with def methods
doesn't flag violation for return inside define_method
behaves like examples with def methods
doesn't flag violation for return inside lambda
behaves like examples with whitelisted method
doesn't flag violation for return inside loop
behaves like examples with whitelisted method
doesn't flag violation for return inside times
behaves like examples with whitelisted method
doesn't flag violation for return inside each
behaves like examples with whitelisted method
doesn't flag violation for return inside each_filename
...
Note the (random)
appendix to each example group to denote that it's run in random order.
TODO YAML
Generated via
find spec ee/spec -type f -name "*.rb" | grep -E "_spec\.rb$|frontend/fixtures" | sort | perl -lne "print qq{- './\$_'}" > spec/support/rspec_order_todo.yml
Current TODOs are:
- All spec files ending with
_spec.rb
- All frontend fixtures (matching
frontend/fixtures
)
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.