Decoupling auto-deploy tagging and rollout
The way release-tools produces and consumes auto-deploy packages is strongly coupled with the type of pipeline that handles this process.
This makes it hard to produce extra packages or to skip certain packages from deployment without affecting the whole pipeline.
When it's time to deploy a new package, we start by producing it. This operation takes almost 1hr. Because the pipeline and the process are designed to consume the package produced by the same pipeline, we wait that time doing nothing.
We should consider splitting tagging and rolling out into two different pipelines. Tagging could run as often as every hour, and rolling out could continue to follow the timeline set by the release managers.
Each coordinated pipeline will be 1hr shorter as the pipeline will select the most recent available package instead of tagging a new one.
Impact on MTTP
We can split MTTP into its components:
- Merge to Tag
- Tag to Package
- Package to Promotable
- Promotable to Production
This change will only affect the Merge to Tag time. Today we have a minimum of 1hr (the pipeline execution time) up to 4hrs in case the pipeline completes just after the auto-deploy branch preparation, and the next one will be in 3hrs (in a day we have 3 spaced out by 2hrs and 6 by 3hrs).
With this change, we could bring it down to 2hr in the worst case, keeping the minimum 1hr for the best case.
Our ability to promote a package will still affect the final MTTP, but it will reduce wasted time at the beginning of the process.
Impact on our deployment tools
This change is an important investment in our ability to expand our deployment capabilities. Splitting tagging and rolling out will allow us to expand those features independently (i.e. experimental packages, ephemeral builds)
With more packages that time to deploy them and delayed pipeline checks for pick-into auto-deploy, each rollout pipeline will have something to deploy. Moreover, we could expand our package selection strategy to consider minimum SHA included in the next package or marking packages as not good for deployment.
The split will also simplify our ability to recover from a release-tools bugs without the need to build a new package.