Document automatic test splitting
Problem to solve
We have the parallel keyword which gives control and flexibility to set up parallel tests (or run any job in a parallel fashion), but this requires a lot of setup from a developer and sometimes the split logic just get duplicated. There are open source solutions like https://github.com/renderedtext/test-boosters/ that take this a step further by also separating the test configuration into multiple files, handling that part of the configuration for you.
Test Boosters is an open source project for helping make test parallelization easy. Test Boosters supports RSpec, Cucumber, Minitest, ExUnit, and Go.
On the GitLab side, we recently added the
parallel keyword (https://docs.gitlab.com/ee/ci/yaml/#parallel) which works as follows:
test: parallel: 5
We can build on top of this by showing people how to use test boosters with this feature.
For the MVC we will add instructions to the official documentation on how to use test boosters with the parallel keyword to achieve better, easier results. The syntax would be something like the following:
test: parallel: 5 script: - gem install semaphore_test_boosters - rspec_booster --job $CI_NODE_INDEX/$CI_NODE_TOTAL
This would create parallel 5 jobs, each running
rspec_booster --job n/5 (where n is the job index)
Permissions and Security
This should not impact the GitLab security model apart from what's already implemented in the
- We should add documentation at the parallel keyword, and also include an example in the quick start guide.
- We should add a ci.yml template that demonstrates how to do it
- We should write a blog post about it (this is a task for the PM)
What does success look like, and how can we measure that?
Success is helping our users set up their parallel tests even easier than before. We can measure # of users who are installing the booster gem as part of their configuration.
Links / references
- Setup guide for test-boosters on Semaphore: https://semaphoreci.com/docs/setting-up-boosters.html
- As discussed during Cancun CI / CD brainstorming session
- CircleCI uses a bespoke CLI for it: https://circleci.com/docs/2.0/parallelism-faster-jobs/