Versioning charts in order to ship breaking change safely
TODO
Phase 1: Prepartion
-
Evaluate PoC for chart migration => !86 (closed) -
Document backport flow => !88 (merged) -
Update .gitlab/issue_templates/Configure upgrade checklist.md
=> https://gitlab.com/gitlab-org/configure/general/-/merge_requests/18 -
Update README.md in auto-deploy-app => gitlab-org/charts/auto-deploy-app!82 (merged) -
Announce in Issues/MRs in auto-deploy-app that the project will be merged into auto-deploy-image soon. => gitlab-org/charts/auto-deploy-app!83 (closed) -
Make an announcement in #whats-happening-at-gitlab
channel about the migration.
Phase 2: Migration
-
Create a pre-release branch in auto-deploy-image. => !90 (merged) -
Copy all files from auto-deploy-app into vendor/auto-deploy-app-chart
of auto-deploy-image. => !90 (merged) -
Ensure test for charts are run on the auto-deploy-image's CI. => !90 (merged) -
Fix Dockerfile
to copy theassets/auto-deploy-app-chart
into the auto-deploy image. => !95 (merged) -
Fix download_chart
function to move the bundled chart for the actual usage (if user-defined chart and chart repository variable do not exist). => !95 (merged) -
Implement version detection => !95 (merged) -
Add doc for public API => !87 (merged) -
Re-run !90 (merged) for making sure everything is synchronized between auto-deploy-image and auto-deploy-app. -
Release v1.0.0 auto-deploy-image => !100 (merged) -
Move all issues to auto-deploy-image
fromauto-deploy-app
=> gitlab-org/charts/auto-deploy-app!84 (closed) -
Close MRs in auto-deploy-app that charts are moved to auto-deploy-image. They should develop in the new place unless they have to release a new chart for v0.x.x auto-deploy-image. => gitlab-org/charts/auto-deploy-app!84 (closed)
Phase 3: Post-migration
-
Create an issue for discussion when we bump the default major version in Jobs/Deploy.gitlab-ci.yml
. (Milestone is tentatively set to GitLab 14.0). => gitlab-org/gitlab#232788 (closed)
Problem
Today, in AutoDevOps/AutoDeploy, the latest released chart is always fetched and used. This sometimes makes a difficult situation when we introduce a breaking change as it interferes the existing setups of users/customers.
For example, in the feature request of Canary Ingress support, we need to introduce a architectural change to the k8s-resources. This likely breaks a backward compatibility, however, we still want to ship the feature as a new standard and slowly migrate users from old version to new version.
Proposal
Originally proposed by @tkuah at gitlab-org/charts/auto-deploy-app#70 (comment 357043316).
Bundle charts (auto-deploy-app) into auto-deploy-image
- We bundle the charts into auto-deploy-image and maintain charts and deploy scripts in the same semantic versioning.
- We deprecate this auto-deploy-app project. Instead, we maintain the charts in auto-deploy-image project.
- We no longer release new charts to charts.gitlab.io. The currently published charts are remaining in charts.gitlab.io in order to support the existing AutoDevOps setups.
-
auto-deploy-image
hasdownload_charts
command. From now on, it basically uses a bundled chart and doesn't download fromcharts.gitlab.io
. - The auto-deploy-image detects the major version difference between previous deployment and on-going deployment. If there is a difference, it fails the deployment job and shows a instruction to users. The instruction has two paths 1) Manual upgrade 2) Keep using the legacy version. See more gitlab-org/charts/auto-deploy-app#70 (moved).
- We provide a manual migration plan for users to upgrade their AutoDevOps setups to the latest version. The manual upgrade guide could be different per change.
- We document the compatibility charts between GitLab-Rails and auto-deploy-image (Currently, v13.1 and v0.17.0)
Jobs/Deploy.gitlab-ci.yml)
Auto Deploy pipeline (- We can bump the minor/revision version of
auto-deploy-image
anytime as it's backward compatible (e.g. v0.1.0 to v0.3.0) - We can bump the major version of
auto-deploy-image
if it met the following criteria:- There is a strategic reason to introduce a breaking change (e.g. There is a new feature that brings a lot of values).
- We've provided a documentation for users to upgrade their AutoDevOps setups. See more gitlab-org/charts/auto-deploy-app#70 (moved)
- Users can stay with the legacy version if they do not want to upgrade immediately.
- We've announced to users about the upcoming breaking prior to the MR lands to gitlab.com.
Pro
- The chart will always be in sync with the calling bash code.
- One less project to maintain
- No need to publish chart anywhere
- No separate versioning scheme, just one version for
auto-deploy-image
going forward. - Users can still use their own charts
Con
- Forking a folder inside
auto-deploy-image
is not straightforward
Compatibility
GitLab-Rails | auto-deploy-image | auto-deploy-app | Comment |
---|---|---|---|
v10.0 ~ v14.0 | v0.X.X | v0.X.X | v0 is deprecated. Transition period to let users upgrade to v1.0.0. |
v13.2 ~ | v1.X.X | N/A (bundled) | The latest stable |
V0 resource architecture
graph TD;
subgraph gl-managed-app
Z[Nginx Ingress]
end
Z[Nginx Ingress] --> A(Ingress);
Z[Nginx Ingress] --> B(Ingress);
subgraph stg namespace
B[Ingress] --> H(...);
end
subgraph prd namespace
A[Ingress] --> D(Service);
D[Service] --> E(Deployment:Pods:app:stable);
D[Service] --> F(Deployment:Pods:app:canary);
D[Service] --> I(Deployment:Pods:app:rollout);
E(Deployment:Pods:app:stable)---id1[(Pods:Postgres)]
F(Deployment:Pods:app:canary)---id1[(Pods:Postgres)]
I(Deployment:Pods:app:rollout)---id1[(Pods:Postgres)]
end
Proposal: V1 resource architecture
graph TD;
subgraph gl-managed-app
Z[Nginx Ingress]
end
Z[Nginx Ingress] --> A(Ingress);
Z[Nginx Ingress] --> B(Ingress);
subgraph stg namespace
B[Ingress] --> H(...);
end
subgraph prd namespace
A[Ingress] --> J(Canary Ingress);
J(Canary Ingress) --> X{should route to canary?}
X{should route to canary?} --> |No| D(Service);
X{should route to canary?} --> |Yes| K(Service);
X{should route to canary?} --> |Yes| L(Service);
subgraph stable track
D[Service] --> E(Deployment:Pods:app:stable);
end
subgraph canary track
K[Service] --> F(Deployment:Pods:app:canary);
end
subgraph rollout track
L[Service] --> I(Deployment:Pods:app:rollout);
end
E(Deployment:Pods:app:stable)---id1[(Pods:Postgres)]
F(Deployment:Pods:app:canary)---id1[(Pods:Postgres)]
I(Deployment:Pods:app:rollout)---id1[(Pods:Postgres)]
end
User flow when there is a major version change
graph TD
A[AutoDeploy Project with v0 image] --> B(Create Pipeline)
C[AutoDeploy Project with v1 image] --> B(Create Pipeline)
B --> D{Is the major version newer than a **previous deployment**?}
D -->|Yes| E[Pipeline fails with the following warning]
D -->|No| F[Pipeline succeeds]
E --> G{Do you want to keep using the legacy version?}
G -->|Yes| H[User overrides the default to v0 image]
G -->|No| I[User follows manual upgrade guide]
Old proposal
## ProposalWe should do versioning charts, and support fetching a specific version from https://charts.gitlab.io/.
For example,
helm fetch ${auto_chart} --untar # If nothing specified, it fetches the latest one.
helm fetch ${auto_chart} --untar --version=0 # It fetches the latest chart of version 0.
helm fetch ${auto_chart} --untar --version=0.6.1 # It fetches the specific chart of version 0.6.1.
helm fetch ${auto_chart} --untar --version=1 # It fetches the latest chart of version 1.
helm fetch ${auto_chart} --untar --version=1.1.1 # It fetches the specific chart of version 1.1.1.
The version should be matched with the git-tags in auto-deploy-app
repository. Currently, the latest version is v0.6.1, so next breaking change should be introduced with major version increment, which is v1.0.0
. Basically, there are no backward compatibility between major version differences. If a user wants to use v1.0.0 chart to a deployment with v0.6.1, they have to delete the current resources at first (otherwise, encountering an error like this), this is subject to document or discuss in a next step.
This allow us to easily extend auto-deploy-image. For example,
function download_chart() {
...
if [[ ! -d "$auto_chart" ]]; then
local chart_version
if [[ -z "$AUTO_DEVOPS_CHART_VERSION" ]]
chart_version='0' # This is the latest stable version of auto-deploy charts. It's compatible with the all existing AutoDevOps projects.
else
chart_version=$AUTO_DEVOPS_CHART_VERSION # If predefined variable is specified, the chart version is used.
fi
helm fetch ${auto_chart} --untar --version=${chart_version}
fi
Users can choose a chart version via the environment variable CHART_VERSION
, which can be defined anywhere like .gitlab-ci.yml or project setting.