CI: Use ephemeral clusters for MR review apps to smoke test upcoming versions of Kubernetes
Summary
As a result of gitlab-org/distribution/team-tasks#1360 (closed), let's start outlining action items we can take to adjust our Charts CI workflow for the following goals:
- Catch up with the Kubernetes release cycle (RE: gitlab-org&11331)
- Enable us to test against new versions of Kubernetes shortly after they're released
- Increase the feedback loop with faster MR pipelines
- Reduce infrastructure cost and shift away from "pet" clusters as much as possible
We can lay out the plan in phases.
Phase 1
-
Implement ephemeral clusters for MR pipelines
With !3378 (merged) we leverage vcluster
to spin up ephemeral clusters backed by k3s
.
This gives us great isolation between environments for different branches, and lets us very easily test against new versions of Kubernetes without the lengthy process of provisioning a cluster with a cloud provider, installing dependencies like Cert Manager and External DNS, and connecting the environment to our CI configuration.
Phase 2
Related to gitlab-org/distribution/team-tasks#1413
Somewhat blocked by gitlab-org/distribution/team-tasks#1445
Once ephemeral cluster review environments are running in MR pipelines, we should be able to remove the previous review app jobs that were deploying to our "pet" clusters that also run the full QA suite.
The goal here is to still fully validate the codebase but focus those longer, expensive tests on our master
and stable
branches.
The fact is MR pipeline review apps currently only deploy a mostly default instance of the Helm Charts, meaning that the review app often does not actually reflect the changes made in the MR. As a result, we end up spending significant time debugging transient pipeline failures that rarely catch true problems with the change set.
I'm going to keep this issue focused per its title: to implement vcluster
for MR pipelines. This has been implemented as of !3378 (merged). To address the forward-looking comment here, that warrants further discussion to be tracked in gitlab-org/distribution/team-tasks#1413 and gitlab-org/distribution/team-tasks#1466.