## Summary Engineers deploying GitLab.com wait far longer than they should because GKE node pools cannot scale up on demand, with a single deploy job recently taking 45 minutes. This issue covers agreeing a node pool configuration, tuned existing pools or a new pool on a different machine family, that removes those waits, and getting that recommendation in front of the GCP contract negotiation while it can still influence pricing. ## Background Two distinct cluster autoscaler failures are blocking deploys 07/08/2026. Scale-ups on N4 fail against a [vCPU quota limit](https://cloudlogging.app.goo.gl/oqMWLdC6VwifpTir9), and scale-ups on N4D fail against [zonal stockouts](https://cloudlogging.app.goo.gl/HLFq44EUwfn7qvWG9). In both cases pods stay pending while the autoscaler retries other node groups, and the deploy job simply waits. We scaled down in [zone b](https://gitlab.com/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/merge_requests/5650) and [zone d](https://gitlab.com/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/merge_requests/5651) `maxSurge` recently back to 20% \~2 weeks ago for `web` main stage workloads, which did not relieve the pressure, so the binding constraint is node availability, not the size of our own workload. **This is observed on 3 consecutive deploys to grpd on 07/08/2026** * `19.3.202608070407` https://ops.gitlab.net/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/jobs/25807767 slowed down at zone b, [n4d stockouts data](https://cloudlogging.app.goo.gl/4YjRRkAib7DrBXTo9) * `19.3.202608070606` https://ops.gitlab.net/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/jobs/25810712 slowed down at zone c, [n4d stockouts data](https://cloudlogging.app.goo.gl/Zw5Q8R9XfmYJE7tE8) * `19.3.202608070708` https://ops.gitlab.net/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/jobs/25812027 slowed down at zone d, [n4d stockouts data](https://cloudlogging.app.goo.gl/SmBX62NccjcjuBH57) There is a second reason to settle the machine family question now. Finance is [negotiating a new GCP contract](https://gitlab.slack.com/archives/C05KWUER6SV/p1786096909765769) expected to be signed in \~2 months, and the per-SKU discount is fixed for the life of that contract once signed. Which machine families we intend to run on is therefore a decision we can still influence, and the recommendation is worth more before signing than after. `pguinoiseau`'s [earlier notes](https://gitlab.slack.com/archives/C0ACE4T2R6W/p1785798120335469?thread_ts=1785779565.749149&cid=C0ACE4T2R6W) on [node type selection](https://dashboards.gitlab.net/d/pgq4qg4/pguinoiseau3a-gke-nodes-analysis-wip) are the starting point for that recommendation. Fleet Management owns the node pools, so the outcome is a plan agreed with them rather than a unilateral change. We want to keep the following things in mind during implementation: 1. A candidate machine family must have both quota headroom and physical capacity in the zones the affected clusters run in. 2. The resulting configuration must hold at full scale and also allow us to complete planned `maxSurge` increase at https://gitlab.com/gitlab-com/gl-infra/delivery/-/work_items/22262. 3. The node type recommendation must be available to the GCP contract negotiation before it is signed. ## Proposed Actions - [ ] Review Pierre's notes/recommendations on node type selection (referenced in [slack thread](https://gitlab.slack.com/archives/C0ACE4T2R6W/p1785798120335469?thread_ts=1785779565.749149&cid=C0ACE4T2R6W)) - [ ] Asses if shrinking RAM/CPU requesets/limits help with placing more pods on the existing set of nodes. - [ ] Asses if existing node pools can be tuned, [some details](https://gitlab.com/gitlab-com/gl-infra/delivery/-/work_items/22068#note_3563626763) of all node pools being used during a deployment, and - [ ] Assess whether adding a new node pool is the right path forward and the the node type. - [ ] Coordinate with the Fleet Management team on the plan. - [ ] Goal should be to return the baseline deployment time per zonal cluster \~20 mins ([example](https://ops.gitlab.net/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/jobs/25812026)) ## References - [vCPU limit logs](https://cloudlogging.app.goo.gl/oqMWLdC6VwifpTir9) - [N4D stockout logs](https://cloudlogging.app.goo.gl/HLFq44EUwfn7qvWG9) - [Slow deployment job example](https://ops.gitlab.net/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/jobs/25810712) - [Related Slack discussion](https://gitlab.slack.com/archives/C05KWUER6SV/p1786096909765769)