Draft: Kubernetes executor: decline job requests when a ResourceQuota has no room

What does this MR do?

Adds an experimental, opt-in capacity check to the kubernetes executor. Before each job request, the provider reads a ResourceQuota in the job namespace and declines the request when the quota has no room for another pod. The runner core then skips the request, and the job goes to another runner with the same tags.

Why

docker-machine and docker-autoscaler only request a job when they have somewhere to put it. The kubernetes executor requests up to concurrent and finds out afterwards whether the pod can be placed. If it cannot, the job waits for a node until poll_timeout and then fails. GitLab has no requeue, so a taken job has to be served.

concurrent is per replica. With maxSurge: 100% and old replicas draining for hours, several sets of replicas place pods on the same node pool and nothing ties their budgets together. The quota counts every pod in the namespace, including those of draining replicas and other managers, so it works as the shared counter.

Tracking and measurements: gitlab-com/gl-infra/production-engineering#29733

The margin

The check is used + in-flight + margin < hard, where in-flight is this manager's accepted jobs that have no pod yet.

The margin is there because managers do not coordinate. Each one reads the same quota and reserves locally. When a slot frees, every manager sees it in the same second and each one may request a job for it. Only one of those jobs fits. Without a margin the others create pods that admission rejects, and those jobs fail. The first version of this had no margin and lost 5 of 30 jobs to that in the sandbox.

How big the overshoot can get: each manager has at most request_concurrency job requests in flight, and each of those passed the check. So the most that can be requested against one freed slot is replicas × request_concurrency. Draining replicas stop requesting, so drains in progress do not add to it. A rollout does: with maxSurge: 100% the new ReplicaSet is Ready before the old one is told to drain, so for a minute or two both are active. That doubles it.

For 3 replicas and request_concurrency = 4 the margin is 24. This is an upper bound. Every request slot on every replica would have to be free and hit the check inside the same second. The sandbox saw 1 to 3 pods of overshoot against a margin of 12. Once there is production data on the quota_used gauge, the margin can come down to the observed peak plus one.

The margin is capacity that is not used. In exchange the runner has nothing to clean up: the quota is maintained by the API server. !7451 (closed) is the alternative that reserves exactly, with one Lease per job, and has to renew and sweep them.

Requirements

The Role needs get on resourcequotas in the job namespace. The quota's hard should be the fully scaled capacity of the node pool, so that pods inside the margin can still be placed:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: capacity
  namespace: runner-workloads
spec:
  hard:
    pods: "240"

[runners.kubernetes.capacity]
  resource_quota = "capacity"
  margin = 24

requests.memory and requests.cpu in the quota are checked too when present, with the per-pod amount taken from the configured pod-level requests. A requests.memory quota makes admission reject any pod in the namespace without a memory request, so it only fits namespaces where every pod has one.

Still to do

• Docs