FY21 Q4 Delivery OKR: Coordinate deploys from release-tools to support multi pipelines => 50% (1/3)

Status 2021-02-02

With a significant amount of tech debt addressed the key stages of the deployment pipeline are close to being coordinated from a single orchestrator (release-tools).

1. A checkmode pipeline to allow for changes to be safely made without affecting ongoing deployments is complete.
2. To move release-tools to being the coordinator for the deployment Prepare and Build phases:
   • The first task of a deployment, tagging, in order to trigger a deployment pipeline is now being initiated from release-tools
   • Deployments will support multi-pipelines by waiting on builds before beginning the deployment. Coordinating this wait has been moved to release-tools and is in review
   • Following this the deployment trigger can be moved to release-tools and phase 1 of creating a single deployment pipeline will be complete
3. To allow Release-tools to become the coordinator for the deploy tasks:
   • Work to remove the complexity of environments, replace with variables is in progress. Once completed, release-tools will be responsible for triggering the deploy to each environment based on results from the previous step.
   • Following this, the Kubernetes deploy tasks can be separated away from the VM deployment and coordinated by the release-tools.

The current state of the pipeline can be seen in #20 (comment 500406034)

Key Results

• Move the release trigger from Omnibus to release tools.
• Single coordination pipeline to trigger independent Prepare, Build and Deploy stages.
  • Prepare
  • Build
  • Deploy
• Create the ability to run release Preparation in a check mode to allow for pipeline testing.

In this OKR we’ll address release complexity in order to support multi pipelines and rollbacks by creating a centralized release controller to initiate and coordinate the Prepare, Build, and Deploy stages using rules. Deployment pipelines will be context aware.

Deployment rules make completing this work a prerequisite for supporting rollbacks.

As a KPI for Delivery, a lower or no change to MTTP should be an indicator of a successful transition.

Desired State

Release Tools centrally coordinates the individual build tasks before initiating the deployments in parallel.

graph LR
 subgraph Prepare
  A(Release Tools)
 end 

 subgraph Build
  A(Release Tools) --> B(Linux Packages) 
  A(Release Tools) --> C(Container Images)
  B --> |Ready?| A
  C --> |Ready?| A
 end

 subgraph Deploy
   A --> D(VM deployment)
   A --> E(K8s deployment)
   D --> |Completed?| A
   E --> |Completed?| A
 end 

Initial State

Currently, the GitLab release pipeline is a single pipeline covering prepare, build and deploy stages all initiated by omnibus tagging. There is no explicit interaction with "Container Images" and Kubernetes deploy is fire and forget.

graph LR
 A(Release Tools) --> B(Linux Package) 
    B --> C(Deployer)
    C --> A
    C --> D(Kubernetes)   

As our pipelines become more complex and the number of supported satellite projects has grown this approach became more complicated to operate and maintain. New services must be built into the existing pipeline, introducing unmanageable complexity. A failing pipeline might not necessarily mean that every stage has failed to execute and so some changes may be deployed without any way of communicating this.

Finally, the current pipeline complexity makes it very difficult to test that pipeline changes have worked as expected as changes must pass through an entire release cycle (1 month) in order to be validated for all release types. Recently this has led to several issues including delivery#1176 (closed) and delivery#1298 (closed).