Configuration Deprecation process Refinement for SaaS
What is the GitLab engineering productivity problem to solve?
Recently, Auto-Deploys became blocked at the turn of the milestone when Omnibus installs began to fail due to configurations that were in place that were considered deprecated a few releases prior. This situation caused Auto-Deploy to become blocked which is a hinderance to the overall Engineering department. In reaction to this, an Incident was created since we were blocked and refinement of the existing procedure to detect deprecations had begun.
During discussion of refining the existing procedure from the perspective of teamDelivery, we discussed what it would be like if we attempted to push left, the desire to test and validate these style of deprecations on GitLab.com prior to release. Since we leverage .com as our testing ground for feature changes prior to release and we have extensive procedures for rolling these changes out on .com for appropriate testing, it would seem we have a gap for configuration changes.
This issue is to have the conversation to decide if we should modify our procedures for configuration deprecations where they are fully tested on .com prior to release. Doing so proactively prevents Auto-Deploy from being blocked and shifts the entire responsibility of managing deprecations from SRE's onto the Development teams introducing the configuration change. This proposal is to modify our existing deprecation policy to become similar to that of Feature Flag rollouts for SaaS.
References:
- Existing "procedure" can be found here
- The Incident: gitlab-com/gl-infra/production#9091 (closed)
- teamDelivery Process Refinement: gitlab-com/gl-infra/delivery#3750 (closed)
Current Situation
- Engineering creates change which is already required to be backward compatible which leverages a new configuration style/method
- Changes are made to Omnibus/Helm charts to bring in the new configuration style/method
- Code is Released
Proposal
- Engineering creates changes that are backward compatible
- Changes are made to Omnibus/Helm charts to bring in the new configuration style/method
- Engineering creates and manages the rollout of the new configuration style/method
- Code is released
Proposed Changes
- Create new issue template similar to that of Feature Flag changes
- Update deprecation MR template - https://gitlab.com/gitlab-org/gitlab/-/blob/9b150d1d044fba93fcf88369165bbe9b77188091/.gitlab/merge_request_templates/Deprecations.md
- Update developer documentation - https://gitlab.com/gitlab-org/gitlab/-/blob/9b150d1d044fba93fcf88369165bbe9b77188091/doc/development/deprecation_guidelines/index.md
- Update handbook product documentation - https://about.gitlab.com/handbook/product/gitlab-the-product/#deprecations-removals-and-breaking-changes
Milestone
-
Discuss! -
Update various bits of docs -
Advertise procedure changes to all of Engineering
Status 2023-04-24
- Issue created
- Next - Refinement