Upgrade GKE clusters to v1.35

Pierre Guinoiseau
2026-04-02T09:19:41Z

## Overview

Kubernetes v1.35 for GKE is now available in the Regular branch. We need to test and roll out this version progressively across our GKE clusters in a controlled manner before auto-upgrades begin in April (so anytime now).

GKE v1.35 notably contains a fix for Cilium that we want to rollout to production ASAP: https://gitlab.com/groups/gitlab-com/gl-infra/-/work_items/1773#note_3126402611

## Risks

Uncontrolled auto-upgrades to v1.35 could introduce:
- API deprecations preventing workload deployments
- Workload crashes due to compatibility issues
- Networking issues within clusters
- Other unforeseen breaking changes

## Upgrade Process

### Control Plane Upgrade

1. Bump the control plane version in Terraform ([example MR](https://ops.gitlab.net/gitlab-com/gl-infra/config-mgmt/-/merge_requests/9893))
2. Allow up to 30 minutes for Terraform to apply

### Node Pool Upgrades

For each node pool, run:
```sh
gcloud container clusters upgrade $cluster --project=$project --location=$location --node-pool=$node_pool --async
```

Node pool upgrades can take several hours on `ops-gitlab-gke` and `gprd-*` clusters. Alternatively, allow node pools to auto-upgrade during the next maintenance window following the control plane upgrade.

### Guidelines

- Change Requests (CRs) are advised for `gprd` clusters
- Non-production clusters can be upgraded in parallel
- Work can be distributed across multiple team members

## References

- [GKE Release Schedule](https://docs.cloud.google.com/kubernetes-engine/docs/release-schedule#schedule-for-release-channels)
- [GKE Release Notes](https://docs.cloud.google.com/kubernetes-engine/docs/release-notes#February_11_2026)
- [Kubernetes v1.35 Release Notes](https://kubernetes.io/blog/2025/12/17/kubernetes-v1-35-release/)
- [Previous v1.34 Upgrade Issue](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/28159)

## Exit Criteria

All of the following GKE clusters are successfully upgraded to v1.35:

- [ ] `db-benchmarking-gitlab-gke` (`gitlab-db-benchmarking`, `us-east1`)
- [ ] `ops-central` (`gitlab-ops`, `us-central1`)
- [ ] `ops-gitlab-gke` (`gitlab-ops`, `us-east1`)
- [x] `pre-gitlab-gke` (`gitlab-pre`, `us-east1`)
- [ ] `gprd-us-east1-b` (`gitlab-production`, `us-east1-b`)
- [ ] `gprd-us-east1-c` (`gitlab-production`, `us-east1-c`)
- [ ] `gprd-us-east1-d` (`gitlab-production`, `us-east1-d`)
- [ ] `gitlab-3okls` (`gitlab-production`, `us-east1`)
- [ ] `gprd-gitlab-gke` (`gitlab-production`, `us-east1`)
- [ ] `release-gitlab-gke` (`gitlab-release`, `us-east1`)
- [x] `gstg-us-east1-b` (`gitlab-staging-1`, `us-east1-b`)
- [x] `gstg-us-east1-c` (`gitlab-staging-1`, `us-east1-c`)
- [x] `gstg-us-east1-d` (`gitlab-staging-1`, `us-east1-d`)
- [x] `gitlab-36dv2` (`gitlab-staging-1`, `us-east1`)
- [x] `gstg-gitlab-gke` (`gitlab-staging-1`, `us-east1`)
- [ ] `prdsub-customers-gke` (`gitlab-subscriptions-prod`, `us-east1`)
- [ ] `stgsub-customers-gke` (`gitlab-subscriptions-staging`, `us-east1`)
- [ ] `stgsub-ref-customers-gke` (`gitlab-subscriptions-stg-ref`, `us-east1`)