## Summary This epic aims to migrate all remaining SaaS GKE clusters from Calico to Dataplane V2. ## Problem Our older GKE clusters are using Calico for their Control Plane, while the newer ones are using the new Dataplane V2 (based on Cillium) instead. Dataplane V2 [has a number of advantages](https://cloud.google.com/kubernetes-engine/docs/concepts/dataplane-v2#advantages_of) over Calico, and addresses some issues we have encountered: * production-engineering#16775+ * https://gitlab.com/gitlab-com/gl-infra/observability/team/-/issues/3120+ * https://gitlab.com/gitlab-com/gl-infra/observability/team/-/issues/1985+ We would like to rebuild those older clusters still using Calico to use Dataplane V2 instead in order to address the above issue, and align our tooling around a single technology for easier maintenance, monitoring and troubleshooting. We have [tested and validated Dataplane V2 on the `gitlab-pre` cluster](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/issues/16871), and [the zonal `gstg` clusters](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/16909) for over 2 years, and [the zonal `gprd` clusters are being rebuilt with it as well](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1606). ## Proposal Migrate clusters using Calico to Dataplane V2. This will involve reprovisioning clusters as the networking provider is configured at cluster creation time. We currently estimate that after inventorying workloads each cluster will take a day to migrate. We will follow the below provisional plan. For zonal clusters we can deprovision the cluster before provisioning a new cluster, if there are no stateful workloads that require migration. - [ ] Inventory: - [ ] workloads - [ ] services - [ ] projects - [ ] Provision cluster with Dataplane V2 - [ ] Migrate workloads - [ ] Migrate stateful workload data - [ ] Deprovision existing cluster ## Additional context - Previous `gstg` zonal cluster rebuild: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/16909 - Previous `gprd` zonal cluster rebuild: https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1606 - We will need to sync with Delivery, the release managers and EOC to ensure they are aware of the migrations <details> <summary>Current status of each cluster managed from `config-mgmt`</summary> ```shell > for project in gitlab-{pre,ops,production,staging-1,release,db-benchmarking,subscriptions-{staging,stg-ref,prod}}; echo "# $project" ; gcloud container clusters list --project $project --format json | jq -r '.[] | " \(.name) => \(if .networkConfig.datapathProvider == "ADVANCED_DATAPATH" then "Dataplane V2" else "Calico" end)"'; end # gitlab-pre pre-gitlab-gke => Dataplane V2 # gitlab-ops ops-central => Dataplane V2 ops-gitlab-gke => Calico # gitlab-production gitlab-3okls => Dataplane V2 gprd-gitlab-gke => Calico gprd-us-east1-b => Calico gprd-us-east1-c => Calico gprd-us-east1-d => Calico # gitlab-staging-1 gitlab-36dv2 => Dataplane V2 gstg-gitlab-gke => Calico gstg-us-east1-b => Dataplane V2 gstg-us-east1-c => Dataplane V2 gstg-us-east1-d => Dataplane V2 # gitlab-release dattang1 => Calico dattang2 => Calico release-gitlab-gke => Dataplane V2 # gitlab-db-benchmarking db-benchmarking-gitlab-gke => Calico # gitlab-subscriptions-staging stgsub-customers-gke => Calico # gitlab-subscriptions-stg-ref stgsub-ref-customers-gke => Calico # gitlab-subscriptions-prod prdsub-customers-gke => Calico ``` </details> ## DRI TBD ## Exit Criteria - [ ] All GKE clusters belonging to the GitLab SaaS infrastructure use Dataplane V2 instead of Calico - [ ] `gstg-gitlab-gke` - [ ] `gprd-gitlab-gke` - [ ] `ops-gitlab-gke` - [ ] `db-benchmarking-gitlab-gke` - [ ] `stgsub-customers-gke` - [ ] `stgsub-ref-customers-gke` - [ ] `prdsub-customers-gke` ## Issue Admin ``` /epic &1651 /labels ~"Department::Infrastructure" ~"devops::Production Engineering" ~"group::Foundations" /labels ~"Foundations::Project work" ~"workflow-infra::Proposal" ~"Production Engineering::P3" ```