## Overview This is a retrospective on the [rebuild of the `gprd-us-east1-b` GKE cluster](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/20220), where [several problems were encountered](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/20220#note_2643545530), causing the rebuild to take longer than it should have, and causing some service disruption near the end. Those problems need to be resolved before continuing with the rebuild of the 2 other zonal clusters. ## What went well * The rebuild was eventually successful. * There were no availability issues while the cluster was offline (except towards the end when EMEA traffic was picking up and affecting the apdex of some services on the other clusters). * The Vault related steps worked flawlessly. ## What went wrong (and solutions) ### HAProxy tooling Our tooling for manipulating backend servers in HAProxy, with the [`get-server-state`](https://gitlab.com/gitlab-com/gl-infra/chef-repo/-/blob/master/bin/get-server-state) and [`set-server-state`](https://gitlab.com/gitlab-com/gl-infra/chef-repo/-/blob/master/bin/set-server-state) scripts, is inadequate today: * It doesn't scale well with the number of HAProxy nodes, and number of backends on each node, we have today (77): * Updating the all 11 zonal cluster backends on ~23 nodes of one zone one by one, twice (because of canary backends), with a 1 minute wait time between each to let the other clusters scale, takes several hours. * Updating the script to allow updating more than 1 node each interval would at least help speed it up. * It works using `knife ssh`, and when using a Yubikey with the touch policy enabled (as required) it's easy to forget to touch it every 15 seconds which causes the script to fail half-way and skip nodes, forcing to start all over again. * There is no progress feedback with those scripts. * After setting backend server states to `MAINT` or `DRAIN`, a Chef run updating the HAProxy config and reloading it will reset those backend server states to `UP`, (this is a [known issue](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/issues/12421#note_2405314246)), this caused the Canary backend servers to be re-enabled automatically halfway through the CR, overloading the Canary stage services. **Solution:** for the next clusters, it might be faster and safer to simply drain and shutdown HAProxy on the nodes of the same zone. During APAC hours the remaining nodes in the other 2 zones should be able to handle the traffic load. ### VPC peering causing subnet overlap issues While creating the new cluster, there was a subnet overlap error: ``` │ Error: Error waiting for creating GKE cluster: Conflicting IP cidr range: New subnetwork overlaps with an IP range (172.16.0.16/28) in one of the active peers of an active peer (peering-gstg). ``` We use the same master IPv4 CIDRs `172.16.0.*` [in `gstg`](https://ops.gitlab.net/gitlab-com/gl-infra/config-mgmt/-/blob/2e16d190dd4f372c87fab72a427abf5158eae9ed/environments/gstg/variables.tf#L628-638) and [in `gprd`](https://ops.gitlab.net/gitlab-com/gl-infra/config-mgmt/-/blob/2e16d190dd4f372c87fab72a427abf5158eae9ed/environments/gprd/variables.tf#L476-486), and the `gstg` and `gprd` VPCs are peered together (for some historic reason that might not be valid anymore today?), hence the overlap. GKE might not have checked for overlaps in the past, but [it does today](https://cloud.google.com/kubernetes-engine/docs/how-to/latest/network-isolation#restrictions_and_limitations). **Solution:** the fix was to update the CIDR to `172.16.1.16/28`, which we will have to do with the other clusters too. > [!note] > This CIDR is [actually optional now](https://cloud.google.com/kubernetes-engine/docs/how-to/latest/network-isolation#modify_the_control_plane_access), so leaving it unset could have been an option, though it would have required updating our `gke` module first. > [!note] > This master subnet used to [created in a Google-managed project and VPC peered with our local VPC](https://console.cloud.google.com/networking/networks/details/gprd?project=gitlab-production&inv=1&invt=Ab3fpQ&pageTab=PEERINGS&pageState=(%22peeringTable%22:(%22f%22:%22%255B%257B_22k_22_3A_22_22_2C_22t_22_3A10_2C_22v_22_3A_22_5C_22gke-prod_5C_22_22%257D%255D%22))) with clusters version 1.28 or older ([docs](https://cloud.google.com/kubernetes-engine/docs/how-to/legacy/network-isolation)), but today with version 1.29 or later it is simply [created directly in our VPC if set](https://console.cloud.google.com/networking/networks/details/gprd?project=gitlab-production&inv=1&invt=Ab3fpg&pageTab=SUBNETS&pageState=(%22networkDetailsSubnetworkTable%22:(%22f%22:%22%255B%257B_22k_22_3A_22_22_2C_22t_22_3A10_2C_22v_22_3A_22_5C_22gprd-us-east1-b_5C_22_22%257D%255D%22))). ### GKE maintenance / Kubernetes API server unavailable Some time after successful creation of the cluster and while redeploying all workloads, the nodes were unable to autoscale or to become ready for close to 1 hour. At that same time the cluster was undergoing some maintenance operations (it is unknown what exactly at this stage). The nodes were unable to get into a healthy stage because the `pdcsi-node` pods were crashlooping because of timeouts while connecting to the API server: ``` gce-pd-driver W0722 08:06:46.717654 12 cache.go:293] Error getting node gke-gprd-us-east1-b-generic-4-d97d359e-hrwf: Get "https://10.65.0.1:443/api/v1/nodes/gke-gprd-us-east1-b-generic-4-d97d359e-hrwf": dial tcp 10.65.0.1:443: i/o timeout, retrying... ``` The problem went away on its own once the maintenance operations were finished. ### `k8s-workloads/gitlab-helmfiles` deployment issues * With Dataplane V2, `calico-node-vertical-autoscaler` no longer exists (replaced with Cillium), so those commands were unnecessary to run: ```shell kubectl -n kube-system annotate cm calico-node-vertical-autoscaler meta.helm.sh/release-name=calico-node-autoscaler kubectl -n kube-system annotate cm calico-node-vertical-autoscaler meta.helm.sh/release-namespace=kube-system kubectl -n kube-system label cm calico-node-vertical-autoscaler app.kubernetes.io/managed-by=Helm ``` * However we now modify a `kube-dns` configmap in `gitlab-helmfiles`, so those steps are now necessary to give it ownership of the configmap: ```shell kubectl -n kube-system annotate configmap/kube-dns meta.helm.sh/release-name=kube-dns-extras meta.helm.sh/release-namespace=kube-system kubectl -n kube-system label configmap/kube-dns app.kubernetes.io/managed-by=Helm ``` * There is a chicken-and-egg situation with the KEDA deployment, because the chart values [include a `ClusterTriggerAuthentication` in `extraObjects`](https://gitlab.com/gitlab-com/gl-infra/k8s-workloads/gitlab-helmfiles/-/blob/68ac1924be797d530f0b662c5d6f1c147f99ab15/releases/keda/values.yaml.gotmpl#L79-91), which requires CRDs that are deployed by this same chart.= ``` Error: unable to build kubernetes objects from release manifest: resource mapping not found for name: "keda-mimir-creds" namespace: "" from "": no matches for kind "ClusterTriggerAuthentication" in version "keda.sh/v1alpha1" ``` **Solution:** this `ClusterTriggerAuthentication` object needs to be moved to a custom chart that will be deployed **after** the `keda` chart. ### `k8s-workloads/gitlab-com` deployment issues The `gitlab` release [failed to deploy from scratch](https://ops.gitlab.net/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/jobs/19411323) because of this error: ``` Error: YAML parse error on gitlab/charts/gitlab/charts/webservice/templates/deployment.yaml: error converting YAML to JSON: yaml: line 555: mapping values are not allowed in this context helm.go:92: 2025-07-22 07:31:57.097011262 +0000 UTC m=+1.050101180 [debug] error converting YAML to JSON: yaml: line 555: mapping values are not allowed in this context YAML parse error on gitlab/charts/gitlab/charts/webservice/templates/deployment.yaml ``` The workaround was to deploy it from a local clone with: ```shell GITLAB_IMAGE_TAG=18-3-202507212221-e25d8ecd126 helmfile -e gprd-us-east1-b apply --skip-deps ``` **Solution:** fix the helmfile release to produce valid YAML when the `gitlab-chart-info` configmap doesn't exist. ### Network policy changes with Dataplane V2 While re-enabling traffic to the new cluster there were errors from the `api` and `pages` services which we unable to connect to the internal `registry` service and `internal-api` service respectively. This was because of Network Policies using CIDRs to control traffic to/from those services, which is [no longer supported with Dataplane V2 because it uses eBPF instead of `iptables`](https://cloud.google.com/kubernetes-engine/docs/how-to/network-policy#using_network_policy_enforcement), pod selectors should be used instead. **Solution:** updated the network policies to use pod selectors: https://gitlab.com/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/merge_requests/4601 > [!note] > This same issue was encountered and fixed when the zonal `gstg` clusters, but was missed when planning this rebuild. ## Next * [x] Fix the `keda` release in `gitlab-helmfiles` * [x] Fix the `gitlab` release in `gitlab-com` * [x] Find more information about the Kubernetes API server outage * [x] Updated the CRs for the `gprd-us-east1-{c,d}` GKE clusters with the findings above * [x] Update the [runbook](https://runbooks.gitlab.com/kube/k8s-cluster-rebuild/) once all clusters have been rebuilt.