Iterative delivery strategies toward the ability to deploy experimental code changes
In the last couple of quarters I've been talking about "delivery.next" with different folks, but it never landed into an issue.
This is a collection of deployment and release capabilities that could bring us to the next level of deployments for GitLab.com
This is my attempt to collect the thoughts behind those discussions, together with some supportive diagrams that I have build over time and start sharing those ideas more widely.
Here follows a very high level description of the above blocks, at the end there is a link to a recording briefly explaining the connection between the blocks.
- Split auto-deploy operations
- asynchronous tagging
🏷 ️- Removing tagging and packaging from the coordinator pipeline
- A new schedule will take care of creating a new auto-deploy branch and immediately tag a package
- We could try to aim for a package each hour as a starting point
- rollout
🚀 : Package selector: last build🔍 - With packaging outside of the coordinator pipeline, we will remove active waiting (about 1hr) from each pipeline
- We will deploy to gstg-cny immediately at the beginning of the pipeline
- Packaging failures will not delay the deployment train
- asynchronous tagging
- Triggered packages
📦 - Today each auto-deploy package commits components version on both omnibus and CNG, polluting the security mirrors with many unnecessary branches, commits and tags
- All the packages that we build are already tracked in release/metadata
- It should be possible to trigger a pipeline to both omnibus and CNG and have them produce the desired packages without committing ant tagging the packagers
- It is important to note that auto-deploy packages are relevant only for half a day (the weekend at maximum), each successful auto-deploy package will either be a seed for a monthly release or be superseded by the next package
- Build from default branch
🏗 ️- We do utilize auto-deploy branches for two reasons: stashing fixes (pick into auto-deploy) and committing components versions on packagers
- We should drop usage of auto-deploy branches and trigger package generation from default branches
- This will remove the current limit of maximum an auto-deploy branch each hour
- Ephemeral packages
👻 - Auto-deploy tagging should provide a continuous stream of packages ready for deployment, however release managers should be able to deviate from this and build a package to fix an incident
- A release-manager should be able to direct auto-deploy to a merge request fixing an incident and have the ability to generate a package that is the current deployed packages plus the desired MR
- Package selector: minimum SHA
🛃 - The rollout process should be able to filter packages by a minimum required SHA
- QA isolation
👷 ♀️- We want to be able to run QA on staging/production prior to route external traffic to the new deployment
- Multi-stage deployments
🍱 - Our environment should not be constrained by only two stages (cny and main)
- It should be possible to add additional stages with custom routing strategies
- blue/green deployments 🟦🟩
- We should be able to deploy a full gitlab deployment and only route QA traffic to this new deployment
- If the QA is successful, we will start routing external traffic and remove the old deployment
- Pre-cny stages 🪺
- With more packages than coordinator pipelines it should be possible to pre-deploy package candidates as pre-cny stages and assess their performance
- Package selector: pre-cny health
🌡 ️- The rollout process should be able to select the next package based on pre-cny deployment health
- Ideally we would like to deploy the most recent package that is meeting the SLO
- Drop pick into auto-deploy
🍒 - Experimental packages 🧪
- We should be able to build packages with features not yet merged into the stable branches
- When we build a package, we should also additionally build all the allowed experiments by picking the desired experiment MR on top of the new package being built (more control, more work)
- Alternatively we could just built experimental packages from the MR SHA giving each experiment a time slot regardless of the status of the main stage (less control, less work)
- Deploy experiments 🧫
- Experimental packages should be deployed to dedicate stages in canary and production and feed back metrics to developers working on those experiments
Edited by Alessio Caiazza