Controlled rollout/rollback of AutoDevops images
We currently rollout new AutoDevops images by updating the template in lib/gitlab/ci/templates/Jobs/Build.gitlab-ci.yml
, example patch: !59525 (merged).
This tightly couples the rollout of new images to deployment of gitlab. On gitlab.com this induces risk on several fronts, due to our relatively slow deployment pipeline:
- Lack of incremental rollout: Once the new version of gitlab lands in production, the new image is now being served to all users.
- Slow rollback: Rolling back requires going through a full deploy cycle, which is on the order of hours. This means sustained customer impact. We ran into this during gitlab-com/gl-infra/production#4288 (closed).
It would help a lot to have a way of incrementally rolling out changes such as this one. For example through means of a feature flag.
Alternatively, having a way to quickly mitigate and rollback via configuration (environment variable, gitlab.rb
setting, admin setting, etc.) would help reduce potential customer impact for this kind of change.