Skip to content

Test main branch against Ruby 3 in a scheduled pipeline

Recently we were able to achieve our first green(-ish) build for our Ruby 3 MR 🎉 : the only two job failures came from the upstream, and all downstream pipelines and QA were passing: https://gitlab.slack.com/archives/CGN8BUCKC/p1650463827523419, https://gitlab.com/gitlab-org/gitlab/-/pipelines/520562059.

The next step is to run Ruby 3 pipeline on a regular basis, ideally for every MR.

Plan

To support the deployment of Ruby 3 to GitLab.com, we eventually need to have a green build on the master branch. We think that building every MR against Ruby 3 on top of Ruby 2 is too costly (see estimation below), so instead we will implement a staged approach to testing:

  • Milestone 1: Developers and CI run on Ruby 2 by default, by every 2 hours we run a Ruby 3 pipeline against master. This pipeline is just informative and must not block deployments or disrupt other workflows. Errors should only be communicated to developers working on the Ruby 3 migration.
  • Milestone 2: If we find that these Ruby 3 builds are mostly stable, and we understand the kinds of regressions that occur, we can devise a workflow for developers (separate issue) for how to tackle such failures and extend the audience of these builds to every contributor. At this point it is expected that if a changeset breaks these builds, the commit authors tackle these (with of our help, if necessary).
  • Milestone 3: If we find that this workflow works well enough, and enough contributors have been exposed to Ruby 3 related builds and failures (i.e. confidence is established in keeping these green), we will swap the 2-hour schedule with Ruby 2. At this point, every MR will build against Ruby 3, and Ruby 2 is only tested against every 2 hours, and before deployments. We should consider to have developers swap over their default Ruby to Ruby 3 at this point.
  • Milestone 4: This would coincide with running Ruby 3 in production; we remove the 2-hourly Ruby 2 build and Ruby 3 becomes the new default.

Rough approximation of additional CI costs

Additional cost of running Ruby 2 jobs on 2-hourly scheduled pipelines:

  • We have 94 RSpec jobs
  • RSpec jobs average duration is approximately 25 minutes
  • Cost per hour of runner VM: $0.095
  • Total additional hours per pipeline: 25 * 94 / 60 = 39.1
  • Total additional cost per pipeline: 39.1 * $0.095 = $3.7
  • We have 12 2-hourly scheduled pipelines per day
  • Total additional cost per day: $3.7 * 12 = $44.4
  • Running these duplicated tests for a year (e.g. until 16.x is released) would result in an additional: $44.4 * 365 = $16,206.
Edited by Yannis Roussos