Test main branch against Ruby 3 in a scheduled pipeline
Recently we were able to achieve our first green(-ish) build for our Ruby 3 MR
The next step is to run Ruby 3 pipeline on a regular basis, ideally for every MR.
Plan
To support the deployment of Ruby 3 to GitLab.com, we eventually need to have a green build on the master branch. We think that building every MR against Ruby 3 on top of Ruby 2 is too costly (see estimation below), so instead we will implement a staged approach to testing:
-
Milestone 1: Developers and CI run on Ruby 2 by default, by every 2 hours we run a Ruby 3 pipeline against
master. This pipeline is just informative and must not block deployments or disrupt other workflows. Errors should only be communicated to developers working on the Ruby 3 migration. - Milestone 2: If we find that these Ruby 3 builds are mostly stable, and we understand the kinds of regressions that occur, we can devise a workflow for developers (separate issue) for how to tackle such failures and extend the audience of these builds to every contributor. At this point it is expected that if a changeset breaks these builds, the commit authors tackle these (with of our help, if necessary).
- Milestone 3: If we find that this workflow works well enough, and enough contributors have been exposed to Ruby 3 related builds and failures (i.e. confidence is established in keeping these green), we will swap the 2-hour schedule with Ruby 2. At this point, every MR will build against Ruby 3, and Ruby 2 is only tested against every 2 hours, and before deployments. We should consider to have developers swap over their default Ruby to Ruby 3 at this point.
- Milestone 4: This would coincide with running Ruby 3 in production; we remove the 2-hourly Ruby 2 build and Ruby 3 becomes the new default.
Rough approximation of additional CI costs
Additional cost of running Ruby 2 jobs on 2-hourly scheduled pipelines:
- We have 94 RSpec jobs
- RSpec jobs average duration is approximately 25 minutes
- Cost per hour of runner VM: $0.095
- Total additional hours per pipeline:
25 * 94 / 60 = 39.1 - Total additional cost per pipeline:
39.1 * $0.095 = $3.7 - We have 12 2-hourly scheduled pipelines per day
- Total additional cost per day:
$3.7 * 12 = $44.4 - Running these duplicated tests for a year (e.g. until 16.x is released) would result in an additional:
$44.4 * 365 = $16,206.