Skip to content

Document automatic test splitting

Problem to solve

We have the parallel keyword which gives control and flexibility to set up parallel tests (or run any job in a parallel fashion), but this requires a lot of setup from a developer and sometimes the split logic just get duplicated. There are open source solutions like https://github.com/renderedtext/test-boosters/ that take this a step further by also separating the test configuration into multiple files, handling that part of the configuration for you.

Intended users

Sasha (Software Developer)

Further details

Test Boosters is an open source project for helping make test parallelization easy. Test Boosters supports RSpec, Cucumber, Minitest, ExUnit, and Go.

On the GitLab side, we recently added the parallel keyword (https://docs.gitlab.com/ee/ci/yaml/#parallel) which works as follows:

test:
  parallel: 5

We can build on top of this by showing people how to use test boosters with this feature.

Proposal

For the MVC we will add instructions to the official documentation on how to use test boosters with the parallel keyword to achieve better, easier results. The syntax would be something like the following:

test:
  parallel: 5
  script:
    - gem install semaphore_test_boosters
    - rspec_booster --job $CI_NODE_INDEX/$CI_NODE_TOTAL

This would create parallel 5 jobs, each running rspec_booster --job n/5 (where n is the job index)

Permissions and Security

This should not impact the GitLab security model apart from what's already implemented in the parallel keyword.

Documentation

  • We should add documentation at the parallel keyword, and also include an example in the quick start guide.
  • We should add a ci.yml template that demonstrates how to do it
  • We should write a blog post about it (this is a task for the PM)

What does success look like, and how can we measure that?

Success is helping our users set up their parallel tests even easier than before. We can measure # of users who are installing the booster gem as part of their configuration.

Links / references

Edited by Jason Yavorsky