## Goal Make the kubernetes executor stop requesting jobs when the workloads cluster cannot place another pod, the way docker-machine and docker-autoscaler already do. Split out of https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/29646 ("Concurrency and capacity"). Background from [note 3798992209](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/29646#note_3798992209) onward. ## Why We are rolling out a new gvisor shard. It takes some of the work, possibly more over time. The deployment uses `maxSurge: 100%` so config changes roll out quickly while the old managers drain for hours. Several drains can be in progress at once, and together they can exceed the node capacity several times over.  Each ReplicaSet is budgeted for 240 jobs. On 2026-09-07 two rollouts 22 minutes apart left three of them running at once, 479 jobs together. When that happens the kubernetes executor still accepts the job, and the job waits for a node. Worse, it can fail. The docker executor avoids this by reserving a machine before it accepts a job. The kubernetes executor cannot: the pod is only created after the job has been dequeued. And GitLab has no reservation or requeue, so a job cannot be accepted and put back. Once it is taken, it must be served. Storing the cap in Redis for coordination would work but adds a single point of failure, contention, and an operational burden. So the design has to satisfy: - Once a job is accepted, we must be able to run it, quickly. - We must predict that we cannot, and stop accepting jobs before that. - Jobs must not wait. - Jobs must not fail. This gives us spillover. Other runner capacity picks up the jobs the shard does not take. That is what lets us move workloads to Kubernetes while keeping the docker-machine setup, without impacting existing workloads. ## Approach Add a capacity check to the kubernetes executor provider's Acquire step. It reads how much room is left before the job request and returns `NoFreeExecutorError` when there is none, so the runner core skips the request. The provider already has Acquire and Release hooks, added for pause pods in https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/6334. The source for remaining room is a ResourceQuota in the job namespace. `status.used` against `status.hard` is maintained by the API server, is the value admission compares against, costs one GET, and counts every pod in the namespace, including pods owned by draining replicas and by other managers on the same cluster. That is the shared counter, kept by the cluster we already depend on: no new service, no Redis, no lag from a metrics pipeline, and no deadlock, since quota is admission only and the cluster autoscaler still scales for admitted pods. `hard` is set to what the node cap can hold, so admission never rejects in practice. The quota is there as the counter. This is the opposite of gitlab-org/gitlab-runner#28184, which retries pod creation after admission has rejected it. The job has already been accepted at that point. ## Questions to answer 1. **In-flight accounting.** Several Acquire calls run before any pod exists. The provider has to reserve locally (remaining quota minus this manager's in-flight acquires) and give it back on Release, or one poll cycle overshoots by up to `concurrent`. docker-machine and docker-autoscaler both do this and are the reference. 2. **One pod's worth.** Quota `requests.memory` sums all containers in the pod: build (`memory_request`, or the job's overwrite up to the configured max), helper, and services. Acquire runs before the job is known. Reserve the configured default and let `hard` bound the overshoot from overwrites, or reserve the max allowed? p-gv-s runs 8Gi default, 32Gi max. 3. **Quota freshness.** Admission updates `used` synchronously on create. On delete the quota controller updates it asynchronously. Measure the lag. If it matters, the provider can subtract its own recently deleted pods. 4. **Scoping.** Per-size budgets are possible with one namespace per shard, or one PriorityClass per shard (`priority_class_name`) with a scoped quota each, on top of one shared `requests.memory` quota for the pool. The shared quota mirrors the node pool, which is the resource the sizes share. Per-size quotas only matter if large jobs need guaranteed room against small ones. Decide whether the provider reads one named quota or all quotas that apply to the pod it would create. 5. **Which resource.** `pods` is simple but ignores size. `requests.memory` is what fills a node here. Probably both, with memory as the one that decides. 6. **Config surface.** Something like `[runners.kubernetes.capacity]` with a quota name and an optional per-pod reservation, off by default. Or discover the quotas in the namespace. 7. **Zero room left.** Decline, check again on the next poll (`check_interval`). Is the poll loop enough or is backoff needed? Log line and metric so an operator can tell "declining for capacity" from "idle": a gauge for remaining quota and a counter of declined acquires. 8. **Rollout stacking.** With the check in place, new replicas see the old replicas' pods in `used` and request only into what is left. Confirm on a real rollout on p-gv-s: `gitlab_runner_jobs` summed over old and new pods stays under budget. 9. **Two managers racing for the last slot.** Both see room for one pod and admission rejects the second. The existing `retry_limits` entry for `exceeded quota` retries the pod creation, not the job. Confirm the race is rare and the retry covers it with no effect on the job. 10. **Relation to `concurrent`.** It stays as the per-replica cap protecting the manager itself (about 8 MiB per running job). The quota check is the cluster-level one. ## Prototype - A runner build with the capacity check, deployed as several manager replicas against a cluster with a ResourceQuota in the job namespace. First in the sandbox, then on p-gv-s. - Set the quota below what the replicas would request together and put load on it. The sum of running jobs across replicas stays at the quota, the declined counter increments, the rest of the jobs stay queued on GitLab, and none fail. - Roll the managers during load for question 8. - On p-gv-s the mirror project is the only traffic and nothing else on the cluster changes. ## Out of scope - Failing jobs so GitLab retries them. - Reading Mimir or any metrics pipeline for the decision. - Any new service or shared store. - Reacting to autoscaler events (`NotTriggerScaleUp`). They happen after acceptance. ## Output - Working prototype on p-gv-s with the rollout numbers from question 8. - Answers to the questions above in the notes here. - MR to gitlab-org/gitlab-runner with the docker-machine and docker-autoscaler comparison as the motivation.