Draft: Kubernetes executor: reserve a job slot before requesting a job

What does this MR do?

Adds an experimental, opt-in capacity check to the kubernetes executor. With [runners.kubernetes.capacity] enabled = true, the provider creates a Lease before each job request and declines the request with NoFreeExecutorError when a ResourceQuota on count/leases.coordination.k8s.io rejects the create. The Lease is held until Release, renewed every 30s, and leases nobody renewed for 5 minutes are deleted by whichever manager sweeps next.

Why

docker-machine and docker-autoscaler only request a job when they have somewhere to put it. The kubernetes executor requests up to concurrent and only finds out afterwards whether the pod can be placed. If it cannot, the job waits for a node until poll_timeout and then fails. GitLab has no requeue, so a taken job has to be served.

concurrent is per replica. With maxSurge: 100% and old replicas draining for hours, several sets of replicas place pods on the same node pool and nothing ties their budgets together.

Admission is the only atomic step available across managers, so the reservation has to be an object that goes through admission. Pods would work but the pod does not exist yet at request time, and placeholder pods distort VPA and workload accounting. Reading the quota and reserving locally was tried first (first commit) and races: when a slot frees, every replica sees it and requests a job for it. A Lease is not a workload, goes through quota admission, and is cheap.

The budget is the namespace the leases are created in (capacity.namespace, defaulting to the job namespace), so runners sharing a node pool share one quota. Mixed sizes in one budget are not weighted yet.

Tracking and measurements: gitlab-com/gl-infra/production-engineering#29733

Requirements

The runner's Role needs create, delete, get, list, update on leases.coordination.k8s.io in the lease namespace, and a ResourceQuota there:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: capacity
  namespace: runner-workloads
spec:
  hard:
    count/leases.coordination.k8s.io: "240"

Still to do

• Docs
• Stale window as config rather than a constant
• Weighted leases for mixed sizes in one budget