Advanced deploys (Blue/green, Canary, Traffic vectoring)

Description

Today, GitLab CI believes that deploys are binary, all-or-nothing affairs. But some teams use blue/green deploys, canary deploys, and traffic vectoring to reduce risk during deploys. This can actually take many shapes as people interpret the idea differently to suit their needs.

Blue/green deploys Create two (nearly) identical environments, arbitrarily called blue and green. One isn't better than the other, just separate.

With all traffic going to green, load new code onto blue, get it ready, and then switch the router to suddenly, as quickly as possible, send all new traffic to blue.

Canary deploys, incremental rollouts, traffic vectoring Similar to blue/green deploys, you have a new version of code running at the same time as old code, but instead of cutting over 100% in one moment, you send a portion of traffic to the new version. This can be done in several ways:

1. Given X instances/containers, load the new code on a portion of the instances/containers.
2. or Set the router to send a percent of traffic to the new instances/containers running the new code (traffic ventoring).

The first is easier conceptually and fits well with Docker workflows where it's easy to destroy and create new containers running new images. The second is easier when you have stable instances, are willing to allocate double capacity, or when instance granularity is too large. e.g. if you only have 2 instances in production, the minimum granularity is 1 instance, which is 50% of your capacity. If you want only 10% of traffic to test the new changes, you could use a third instance and send 10% of the total traffic there.

Often when people refer to canary deploys, it's a fixed, small percentage of instances/containers/traffic. Like 10%. Or maybe just 1 instance. But it's possible to extrapolate from there to make an incremental rollout where you start with a small canary deploy, then increase the cutoff gradually, say from 1% to 10% to 25% to 50%, until finally reaching 100%. If at any point, a problem is detected, it should be easy to revert back to 0% to fallback to the old code.

Proposal

Links / references

• Redhat demo of blue-green: https://youtu.be/ooA6FmTL4Dk?t=15m53s
• Redhat demo of canary: https://youtu.be/ooA6FmTL4Dk?t=36m50s
• Kubernetes Deployment Strategies : https://www.weave.works/blog/kubernetes-deployment-strategies

Monitoring stats in merge requests make it easier to see if the change in code or the change in environment/time was the cause.

Multiple canaries need to be in production at the same time if you have many merges on master per day and a long time per canary. But that means that you need to stop all canaries in flight in case of a anomaly. So won't assume that for now. Need to say in MR button: merge & deploy after 2 changes are done.

Canary is branch in repo or revision, feature flags are branch in code path. We will do Canaries after merging into master.

All Canaries should be automatically shifting percentages, 1/5/20/50/100%. Automatically go back to 0, revert the merge, and open an issue if a significant negative change is detected. Or just go 1% per x seconds so that it is easier for the container scheduler to adapt. Also want to look into the Kubernetes tooling that is available.

All Canaries should be tracked with a metric in Prometheus.

added ~333913 label

FWIW, we are already using blue/green deploys with Gitlab. Our environment names include both a color and an actual environment, i.e. production-green

@DanielDent That's great! Any chance you want to share your .gitlab-ci.yml so others can learn how you're doing it?

BTW, this is what IBM does for blue/green deploys on their own CD (using Cloud Foundry):

#!/bin/bash
BLUE_APP="$CF_APP-blue"
GREEN_APP="$CF_APP-green"
cf push "${BLUE_APP}" -n "${CF_APP}"
cf app "${GREEN_APP}" && cf stop "${GREEN_APP}"
cf app "${GREEN_APP}" && cf delete "${GREEN_APP}" -f
cf rename "${BLUE_APP}" "${GREEN_APP}"
export APP_NAME="$GREEN_APP"
export APP_URL=http://$(cf app $APP_NAME | grep urls: | awk '{print $2}')

# Instant Purging removes content from Fastly immediately so it can be refreshed from your origin servers.
# The main entry point http://api.fastly.com/ for the API is not needed for an individual URL Instant Purge.
curl -X PURGE ${APP_URL}

It would be great if in the Environments tab if you could represent which environment is currently served up as the 'live' application. Just a small visual tweak like this to give a quick glance view:

added devopsrelease [DEPRECATED] label

added Category:Continuous Delivery + 1 deleted label

added [deprecated] Accepting merge requests + 1 deleted label

added 2 deleted labels and removed 2 deleted labels

mentioned in epic &494 (closed)

added to epic &770

added Category:Advanced Deployments + 1 deleted label

added maturitylovable + 1 deleted label

changed milestone to %Backlog

added Product Vision FY20 + 1 deleted label

@DanielDent If you don't mind would you share an example of Blue/Green deployment or a template from your gitlab-ci file. That would be really helpful. Thank You.!!

Set /milestone %12.7

changed milestone to %12.7

set milestone to 12.6

changed milestone to %12.6

Added label ~"group::progressive delivery"

added grouprelease [DEPRECATED] + 1 deleted label

Added another reference to the link section of the description

changed the description

added Enterprise Edition + 1 deleted label

marked this issue as related to #14763 (closed)

Added a related issue to improve the documentation to explain how to use blue/green deployments as incremental rollout is already supported and blue/green is a subset within that

added priority3 + 1 deleted label

Changed label in order not to create conflicts with the bug labels

added 2 deleted labels and removed priority3 + 1 deleted label

promoted to epic &2213

promoted to epic so the milestone here is irrelevant follow &2213

changed milestone to %Backlog