Versioning charts in order to ship breaking change safely

TODO

Phase 1: Prepartion

Evaluate PoC for chart migration => gitlab-org/cluster-integration/auto-deploy-image!86 (closed)
Document backport flow => gitlab-org/cluster-integration/auto-deploy-image!88 (merged)
Update .gitlab/issue_templates/Configure upgrade checklist.md => https://gitlab.com/gitlab-org/configure/general/-/merge_requests/18
Update README.md in auto-deploy-app => !82 (merged)
Announce in Issues/MRs in auto-deploy-app that the project will be merged into auto-deploy-image soon. => !83 (closed)
Make an announcement in #whats-happening-at-gitlab channel about the migration.

Phase 2: Migration

Phase 3: Post-migration

Create an issue for discussion when we bump the default major version in Jobs/Deploy.gitlab-ci.yml. (Milestone is tentatively set to GitLab 14.0).

Problem

Today, in AutoDevOps/AutoDeploy, the latest released chart is always fetched and used. This sometimes makes a difficult situation when we introduce a breaking change as it interferes the existing setups of users/customers.

For example, in the feature request of Canary Ingress support, we need to introduce a architectural change to the k8s-resources. This likely breaks a backward compatibility, however, we still want to ship the feature as a new standard and slowly migrate users from old version to new version.

Proposal

Originally proposed by @tkuah at #70 (comment 357043316).

Bundle charts (auto-deploy-app) into auto-deploy-image

We bundle the charts into auto-deploy-image and maintain charts and deploy scripts in the same semantic versioning.
We deprecate this auto-deploy-app project. Instead, we maintain the charts in auto-deploy-image project.
We no longer release new charts to charts.gitlab.io. The currently published charts are remaining in charts.gitlab.io in order to support the existing AutoDevOps setups.
auto-deploy-image has download_charts command. From now on, it basically uses a bundled chart and doesn't download from charts.gitlab.io.
The auto-deploy-image detects the major version difference between previous deployment and on-going deployment. If there is a difference, it fails the deployment job and shows a instruction to users. The instruction has two paths 1) Manual upgrade 2) Keep using the legacy version. See more #70 (moved).
We provide a manual migration plan for users to upgrade their AutoDevOps setups to the latest version. The manual upgrade guide could be different per change.
We document the compatibility charts between GitLab-Rails and auto-deploy-image (Currently, v13.1 and v0.17.0)

Auto Deploy pipeline (Jobs/Deploy.gitlab-ci.yml)

We can bump the minor/revision version of auto-deploy-image anytime as it's backward compatible (e.g. v0.1.0 to v0.3.0)
We can bump the major version of auto-deploy-image if it met the following criteria:
- There is a strategic reason to introduce a breaking change (e.g. There is a new feature that brings a lot of values).
- We've provided a documentation for users to upgrade their AutoDevOps setups. See more #70 (moved)
- Users can stay with the legacy version if they do not want to upgrade immediately.
- We've announced to users about the upcoming breaking prior to the MR lands to gitlab.com.

Pro

The chart will always be in sync with the calling bash code.
One less project to maintain
No need to publish chart anywhere
No separate versioning scheme, just one version for auto-deploy-image going forward.
Users can still use their own charts

Con

Forking a folder inside auto-deploy-image is not straightforward

Compatibility

GitLab-Rails	auto-deploy-image	auto-deploy-app	Comment
v10.0 ~ v14.0	v0.X.X	v0.X.X	v0 is deprecated. Transition period to let users upgrade to v1.0.0.
v13.2 ~	v1.X.X	N/A (bundled)	The latest stable

V0 resource architecture

graph TD;
subgraph gl-managed-app
Z[Nginx Ingress]
end
Z[Nginx Ingress] --> A(Ingress);
Z[Nginx Ingress] --> B(Ingress);
subgraph stg namespace
B[Ingress] --> H(...);
end
subgraph prd namespace
A[Ingress] --> D(Service);
D[Service] --> E(Deployment:Pods:app:stable);
D[Service] --> F(Deployment:Pods:app:canary);
D[Service] --> I(Deployment:Pods:app:rollout);
E(Deployment:Pods:app:stable)---id1[(Pods:Postgres)]
F(Deployment:Pods:app:canary)---id1[(Pods:Postgres)]
I(Deployment:Pods:app:rollout)---id1[(Pods:Postgres)]
end

Proposal: V1 resource architecture

graph TD;
subgraph gl-managed-app
Z[Nginx Ingress]
end
Z[Nginx Ingress] --> A(Ingress);
Z[Nginx Ingress] --> B(Ingress);
subgraph stg namespace
B[Ingress] --> H(...);
end
subgraph prd namespace
A[Ingress] --> J(Canary Ingress);
J(Canary Ingress) --> X{should route to canary?}
X{should route to canary?} --> |No| D(Service);
X{should route to canary?} --> |Yes| K(Service);
X{should route to canary?} --> |Yes| L(Service);
subgraph stable track
D[Service] --> E(Deployment:Pods:app:stable);
end
subgraph canary track
K[Service] --> F(Deployment:Pods:app:canary);
end
subgraph rollout track
L[Service] --> I(Deployment:Pods:app:rollout);
end
E(Deployment:Pods:app:stable)---id1[(Pods:Postgres)]
F(Deployment:Pods:app:canary)---id1[(Pods:Postgres)]
I(Deployment:Pods:app:rollout)---id1[(Pods:Postgres)]
end

User flow when there is a major version change

graph TD
	A[AutoDeploy Project with v0 image] --> B(Create Pipeline)
	C[AutoDeploy Project with v1 image] --> B(Create Pipeline)
	B --> D{Is the major version newer than a **previous deployment**?}
	D -->|Yes| E[Pipeline fails with the following warning]
	D -->|No| F[Pipeline succeeds]
	E --> G{Do you want to keep using the legacy version?}
    G -->|Yes| H[User overrides the default to v0 image]
	G -->|No| I[User follows manual upgrade guide]

Old proposal

## Proposal

We should do versioning charts, and support fetching a specific version from https://charts.gitlab.io/.

For example,

helm fetch ${auto_chart} --untar                 # If nothing specified, it fetches the latest one.
helm fetch ${auto_chart} --untar --version=0     # It fetches the latest chart of version 0.
helm fetch ${auto_chart} --untar --version=0.6.1 # It fetches the specific chart of version 0.6.1.
helm fetch ${auto_chart} --untar --version=1     # It fetches the latest chart of version 1.
helm fetch ${auto_chart} --untar --version=1.1.1 # It fetches the specific chart of version 1.1.1.

The version should be matched with the git-tags in auto-deploy-app repository. Currently, the latest version is v0.6.1, so next breaking change should be introduced with major version increment, which is v1.0.0. Basically, there are no backward compatibility between major version differences. If a user wants to use v1.0.0 chart to a deployment with v0.6.1, they have to delete the current resources at first (otherwise, encountering an error like this), this is subject to document or discuss in a next step.

This allow us to easily extend auto-deploy-image. For example,

function download_chart() {
  ...
  if [[ ! -d "$auto_chart" ]]; then
    local chart_version
    if [[ -z "$AUTO_DEVOPS_CHART_VERSION" ]]
      chart_version='0' # This is the latest stable version of auto-deploy charts. It's compatible with the all existing AutoDevOps projects.
    else
      chart_version=$AUTO_DEVOPS_CHART_VERSION # If predefined variable is specified, the chart version is used.
    fi

    helm fetch ${auto_chart} --untar --version=${chart_version}
  fi

Users can choose a chart version via the environment variable CHART_VERSION, which can be defined anywhere like .gitlab-ci.yml or project setting.

Reference

Edited Jul 29, 2020 by Shinya Maeda