Tests parallelisation using queues (Learned from Shopify)

Listened to a talk about Shopify development stack at Railsclub Moscow. They have team of 300+ devs, 200 servers for CI runners, ~50000 tests, they have integration with Slack very similar to what we show in our demo.

@sytses: what do you think we can learn from Shopify?

First of all, the way they parallelise tests. This is how I understood it: On every build, they boot up 200 Amazon EC2 instances, then they create a queue of 50000 jobs in sidekiq(one job per single test), then this queue gets processed by workers on those EC2 instances.

Obviously, it means that they know the structure of the tests - this is not a technology agnostic way of parallelisation.

But they were able to achieve significant results: all tests are processed in ~5 minutes.

They use Buildkite CI for this.

They have auto-deploy every 30 minutes. You can turn of autodeploy via slash command in Slack in case of some problems.

They have integration with GitHub and Slack. They had to write their Chrome extension which modifies GitHub interface, and notifies users that they shouldn’t merge their branch for master in order not to be deployed (don’t remember the reason for this)

I really like the idea of using a queue of jobs per test instead of lists of commands to execute. I wonder if we could apply this in our product somehow.

The interesting thing is that it would be very valuable feature for big customers, so we could charge a lot for it.