**A note on parallelization** Parallelization is a good short-term solution to quickly reduce the feedback cycle for developers before their contribution can be reviewed and merged. However, its cost can become a problem and ideally any developer should be able to run the full test suite in a resonable time on their own computer. Right now, the full test suite takes 7 hours to finish without parallelization. ### 1. Monitor and profile our RSpec suite Our Redash dashboard will be our source of truth: http://redash.gitlab.com/dashboard/test-suite-statistics - [x] Add `rspec_profiling` to the test environment, run it on `master` and store the data into a new PG database (https://gitlab.com/gitlab-com/infrastructure/issues/976): !7854 - [x] Track EE as well: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/10226 - [x] Exploit this DB using Redash (gitlab-com/infrastructure#877) - Create Redash graphs: - [x] "The 50 most ..." (https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/7854#note_19287450) - [x] Consolidate suite duration per day / week in order to see the progress made (or not): #28777 1. We need to consolidate average duration per day to see the trend go down (or at least stabilize / increase more slowly than today) ### 2. Create project without repos when possible - [x] Convert specs from `create(:project)` to `create(:empty_project)` where possible: - [x] API: !8608 - [x] Use `:empty_project` in factories association: !8770 - [x] Use `:empty_project` where possible in lib specs: !8767 - [x] Use `:empty_project` where possible in finder specs: !8794 - [x] Use `:empty_project` where possible in helper specs: !8796 - [x] Use `:empty_project` where possible in controller specs: !8797 - [x] Use `:empty_project` where possible in model specs: !8832 - [x] Use `:empty_project` where possible in request specs: !8833 - [x] Use `:empty_project` where possible in service specs: !10274 - [x] Use `:empty_project` in worker, view, task, serializer, policy, migration, and mailer specs: !10272 - [x] Use `:empty_project` in feature specs - [x] Add a `:repository` trait so that the `:project` factory doesn't create a repo by default in the future: !8605 - [x] Make the default behavior the fast one (rename `:empty_project` into `:project`. (https://gitlab.com/gitlab-org/gitlab-ce/issues/24899#note_19284116) 3. Investigate bypassing the login procedure for most feature specs - #30196 - [x] Use `sign_in` instead of `login_as` when we're not testing login flow: !10296 - [x] Except where necessary, use `sign_in` over `login_as` in features: !10320 ### 3. Some techniques that can reduce tests duration - [ ] Consider using database deletion over truncation in tests for performance: #30783 / !10624 - [ ] #23036 - Migrate all Spinach tests to RSpec (since we currently only measure RSpec tests duration, our reporting is actually not complete) - [ ] Reduce the main offending test files to reduce their overall duration: http://redash.gitlab.com/queries/15 - [x] #28331 - [ ] #28332 - [ ] #28333 Once we don't have any test file that is very long to execute, we will need to look for particular examples that don't take long compared to the suite duration but that take long compared to what they should be testing. #### Use `before(:context)` / `after(:context)` (with caution!) In some cases, using `before(:context)` / `after(:context)` should be used, for instance in this [particular case][before-context], the test duration was reduced from ~8 minutes to 11 seconds! That's because in that case, the setup was where most of time was spent (~10s) so it made sense to do it only once. [before-context]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/9266#note_23451303 - [x] Use rspec-set to speed up examples: !10551 #### Group expectations Even though in theory we should have one expectation per test, in practice it's often OK to group several expectations (possibly using `aggregate_failures`), particularly when the test setup is heavy. In the same idea you can use the `have_attributes` matcher, so instead of: ```ruby # Here the setup is run twice it { expect(model.name).to eq('foo') } it { expect(model.size).to eq(2) } ``` you can do: ```ruby # Here the setup is run only once it { expect(model).to have_attributes(name: 'foo', size: 2) } ``` #### Chain expectations RSpec allows to use compund matchers, so instead of: ```ruby # Here the setup is run twice it { is_expected.to start_with('Foo') } it { is_expected.to end_with('Bar') } ``` you can do: ```ruby # Here the setup is run only once it { is_expected.to start_with('Noel').and end_with('Rappin') } ``` #### Move some callbacks to the Service layer Callbacks that are only executed on `:create` or on `:update` are good candidates to go into the Service layer for a few reasons: 1. Models shouldn't care about lifecycle-specific behaviors 1. These extra behaviors won't be triggered when using spec factories (at least until we use the services in factories...) meaning less side-effects and a less slower test suite in the end. Examples: ```ruby before_validation :generate_password, on: :create before_validation :signup_domain_valid?, on: :create, if: ->(user) { !user.created_by_id } after_create :post_create_hook ``` from `app/models/user.rb`. #### Move validations to the Service layer – #27100 In the same idea, we could move validations to the service layer to remove side-effects from `Model.create`. Examples: ```ruby validate :namespace_uniq, if: ->(user) { user.username_changed? } validate :avatar_type, if: ->(user) { user.avatar.present? && user.avatar_changed? } validate :unique_email, if: ->(user) { user.email_changed? } validate :owns_notification_email, if: ->(user) { user.notification_email_changed? } validate :owns_public_email, if: ->(user) { user.public_email_changed? } ``` from `app/models/user.rb`. #### Consider removing tests that are duplicated We already have a very good coverage and that probably means that we're over-testing some parts of the application. We might want to remove some tests. ### 4. Don't depend on gitlab-test / external / actual repositories in tests - Consider using Rugged to build repo programmatically instead of relying on gitlab-test: gitlab-org/gitlab-ce#23517 - [ ] We need to abstract the programmatic creation of repos (starting with some helpers) - [ ] We should document how to use these helpers - [ ] We should tell people to use this new strategy by default Prior work: - Test suite statistics: https://gitlab.com/gitlab-org/gitlab-ce/issues/23034 - !6197 - !6722 - !6723 - !6730 - !7282 - !7283 - !7512 - !7779 - !7784 - !7854