## Summary ### Current state Each environment (gstg, gprd) runs **4 GKE clusters** relevant to this proposal, all in `us-east1`. Using gprd as the example: | Cluster | Type | Workloads | |---|---|---| | `gprd-gitlab-gke` | Regional | Sidekiq (all shards), Mailroom, KAS, Zoekt; canary webservice/Shell/Registry in `gitlab-cny` namespace | | `gprd-us-east1-b` | Zonal | Webservice, Shell, Registry, nginx-ingress, Pages | | `gprd-us-east1-c` | Zonal | Same as above | | `gprd-us-east1-d` | Zonal | Same as above | gstg has the same topology, with cluster names prefixed by `gstg-` instead of `gprd-`. User-facing HTTP traffic (web, API, git, websockets, registry, Pages) is served by the **3 zonal clusters**, while background processing (Sidekiq), KAS, Zoekt and Mailroom run exclusively on the **regional cluster**. Canary deployments run in the `gitlab-cny` namespace on the regional cluster but only cover webservice, Shell and Registry — there is no phased rollout for Sidekiq. The [deployment pipeline](https://gitlab.com/gitlab-com/gl-infra/k8s-workloads/gitlab-com) rolls out changes in stages: canary → regional (Sidekiq/KAS) → `gprd-us-east1-b` → `gprd-us-east1-c` + `gprd-us-east1-d` in parallel. The zonal clusters were originally created to keep network traffic zone-local and avoid cross-zone VPC costs. As [documented in the HAProxy runbook](https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/frontend/haproxy.md), each HAProxy node is paired with the zonal cluster in its own zone (active backend), with the other 2 zonal clusters configured as `backup` only. This keeps traffic in-zone for cost savings while still allowing failover during a zonal incident. The `gprd-gitlab-gke` regional cluster still runs Calico and is pending migration to Dataplane V2 per [&1651](https://gitlab.com/groups/gitlab-com/gl-infra/-/work_items/1651). The 3 zonal clusters were recently rebuilt with Dataplane V2 and larger node subnets as part of [&1606](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1606). ### Problem statement 1. **No control plane HA on zonal clusters.** A [zonal GKE cluster has a single-zone control plane](https://cloud.google.com/kubernetes-engine/docs/concepts/regional-clusters) that becomes fully unavailable during upgrades. This blocks deployments and prevents both pod and node autoscaling, requiring careful coordination of maintenance windows across 3 clusters per environment. Google [recommends regional clusters for production workloads](https://cloud.google.com/kubernetes-engine/docs/concepts/regional-clusters) for this reason. 2. **Single-zone blast radius.** A zonal outage takes an entire cluster offline. Pods cannot be rescheduled to other zones because the cluster is zone-scoped. The other 2 clusters eventually scale up in response to the increased traffic, but this is significantly slower than automatic pod rescheduling and the service is severely degraded in the meantime. 3. **No redundancy for Sidekiq, KAS and Zoekt.** All Sidekiq shards, KAS and Zoekt run on the single `gprd-gitlab-gke` regional cluster. If that cluster has a control plane outage or other disruption, background job processing, agent connectivity and code search are all degraded with no failover. The canary stage has the same single point of failure since it also runs exclusively on `gprd-gitlab-gke`. 4. **No phased rollout for Sidekiq, KAS or Zoekt.** Webservice benefits from a staggered rollout across 3 zonal clusters (`gprd-us-east1-b` first, then `gprd-us-east1-c` + `gprd-us-east1-d`), but Sidekiq, KAS and Zoekt are updated all at once on the single regional cluster with no ability to catch regressions before full exposure. 5. **Operational overhead of 4 clusters.** Each cluster needs its own maintenance windows (staggered across the week), upgrade cycles, Terraform state, and network allocations. Major maintenance work — such as rebuilding `gprd-gitlab-gke` for the [Dataplane V2 migration](https://gitlab.com/groups/gitlab-com/gl-infra/-/work_items/1651) — is very difficult to plan and execute because of the lack of redundancy: there is no second cluster to fail over to while the regional cluster is being rebuilt. ### Proposed solution Replace the 3 zonal clusters and the `gprd-gitlab-gke` regional cluster (and the gstg equivalents) with **2 new regional clusters** per environment, each running the **full GitLab Helm chart** (webservice + Sidekiq + supporting services): - **Regional control planes** are [replicated across 3 zones by GKE, upgraded one zone at a time](https://cloud.google.com/kubernetes-engine/docs/concepts/regional-clusters), providing resilience from single zone failure and continuous control plane availability during upgrades — the Kubernetes API remains available and both pod and node autoscaling continue to function - **Worker nodes distributed across zones** — by default, [regional clusters distribute nodes across multiple zones](https://cloud.google.com/kubernetes-engine/docs/concepts/regional-clusters), so a zone failure only loses a fraction of capacity and pods are automatically rescheduled to the remaining zones - **Full chart per cluster** enables phased deployments for **all** workload types: the main stage (webservice, Sidekiq, KAS, Zoekt and all other components) would be deployed to cluster A first, then cluster B, the same way webservice rollouts are staggered across zonal clusters today. Sidekiq, KAS, Zoekt and canary also gain redundancy across 2 clusters instead of depending on a single one - **Zone-aware traffic routing** using Kubernetes Service [`trafficDistribution: PreferSameZone`](https://kubernetes.io/docs/reference/networking/virtual-ips/#traffic-distribution) routes traffic to same-zone endpoints when available, mitigating the cross-zone VPC cost concern that originally motivated the zonal design; fully realising this also depends on the [Envoy Gateway migration](https://gitlab.com/groups/gitlab-com/gl-infra/-/work_items/1641) to replace the current HAProxy and nginx-ingress setup - **Fewer clusters, easier maintenance** — going from 4 to 2 per environment reduces maintenance windows, upgrade coordination, Terraform state, and network allocations. Major operations like cluster rebuilds become tractable because the other regional cluster provides full redundancy during the work. More efficient bin-packing of resources across zones may also slightly reduce costs - **New clusters built right** — the new clusters would be provisioned with Dataplane V2 from the start, addressing [&1651](https://gitlab.com/groups/gitlab-com/gl-infra/-/work_items/1651) for these clusters without a separate migration effort A detailed migration plan will follow _later_. ## References - [GKE Regional Clusters](https://cloud.google.com/kubernetes-engine/docs/concepts/regional-clusters) - [Kubernetes Service `trafficDistribution`](https://kubernetes.io/docs/reference/networking/virtual-ips/#traffic-distribution) - [HAProxy runbook](https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/frontend/haproxy.md) — per-zone HAProxy backend configuration - [&1651 — Migrate SaaS GKE clusters to Dataplane V2](https://gitlab.com/groups/gitlab-com/gl-infra/-/work_items/1651) - [&1606 — Increase GKE node IP subnet size and migrate gprd zonal clusters to Dataplane V2](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1606) - [&1641 — Migrate to Envoy Gateway for GitLab.com](https://gitlab.com/groups/gitlab-com/gl-infra/-/work_items/1641) - [k8s-workloads/gitlab-com](https://gitlab.com/gitlab-com/gl-infra/k8s-workloads/gitlab-com) — Helmfile deployment definitions - [config-mgmt cluster definitions](https://ops.gitlab.net/gitlab-com/gl-infra/config-mgmt) — `environments/{gprd,gstg}/gke-*.tf` _Drafted using Duo_ ~"Powered by Duo"