kubernetes: preemptive mode for pause pods

What does this MR do?

Gives preemptive_mode on [[runners.kubernetes.autoscaler.policy]] a meaning. The field exists on the shared policy config for the instance autoscaler, where it means "request a job only when an instance is confirmed to be available", and was ignored for pause pods. With it set, the runner requests a job only when one of its pause pods is bound to a node. Experimental, and unlike the instance autoscaler off unless set, so existing pause pod users see no change. Otherwise the request is skipped and the job is left in the queue for other runners with the same tags.

• Pause pods scale to zero when the runner begins shutting down. A draining replica kept its pause pods until its last job finished, hours with a long shutdown timeout, while the replacement replicas added their own.
• Pause pods request and schedule like job pods. They request what a job pod with the default settings would, and use the same affinity and scheduler name. Before, they requested only the build container's cpu and memory and could be scheduled where a job pod would not fit. Existing pause pod users will see their pause pods grow by the helper request. Closes #39615 (closed).
• The check before the job request. A pod informer counts the manager's pause pods that are bound to a node. Each job request is counted against them until its job pod has a node or the request ends without a job. The count comes from an informer rather than the Deployment status because another replica's job pod can preempt this one's pause pod, and a 10 s poll would accept one job too many in that window. Policies without preemptive_mode request jobs as before.

Why was this MR needed?

The Kubernetes executor requests jobs before it knows whether it has the capacity to run them. A job starts, then waits for its pod, which may wait for a node. Pause pods were added to over-provision for this, and they help, but they are best effort.

When the node pool is full and the cloud provider cannot add a node:

• The runner still picks the job up, and no other runner can take it. GitLab has no way to hand a job back.
• The job pod is Pending while the cluster autoscaler retries the scale-up and backs off. The customer may be billed for that time.
• After poll_timeout the job fails with runner_system_failure. During a GCE stockout on 2026-10-21 that was 28 jobs in an hour in config-mgmt on the ops runner.
• A longer poll_timeout turns the failures into longer waits and hides the next shortage.

The idea is to use the pause pod as the reservation for a job. The runner schedules pause pods, and only when one is running, which means the capacity exists, does it request a job from the GitLab API. That is backpressure: jobs the cluster cannot place remain in the queue, where other runners with the same tags can take them, instead of waiting in a pod. It is the same model we use in docker+machine and docker-autoscaler: reserve a machine before the request, skip the request when none is free.

sequenceDiagram
    participant R as Runner
    participant K as Kubernetes
    participant G as GitLab

    Note over R,K: today
    R->>G: request job
    G-->>R: job
    R->>K: create job pod
    K-->>R: Pending, no node
    Note over R,K: wait for poll_timeout, fail

    Note over R,K: with preemptive_mode
    R->>K: pause pods (idle_count)
    K-->>R: 2 of 3 bound to a node
    R->>G: request job (2 slots)
    G-->>R: job
    R->>K: create job pod, preempts a pause pod
    Note over R,G: no slot: skip the request, job remains queued

A pause pod bound to a node is room the scheduler has already found for a pod of the job's size and constraints. Using it as the signal means:

• No job is requested for a node that does not exist yet. In a stockout the pause pods go Pending and the runner stops requesting until they are scheduled again.
• A full node pool or a stockout shows up as pending pause pods and stops requests without anyone setting a cap.
• Replicas need no coordination, because each one counts only its own pause pods.

Two other designs for the same check are in !7451 (closed) (a Lease per job behind a ResourceQuota) and !7452 (closed) (a ResourceQuota read with a margin). Both need a cap in the config that matches the node pool, and both accept jobs while the cluster autoscaler is still adding a node.

What's the best way to test this MR?

On a cluster with pause pods enabled and idle_count set so the node pool is full:

1. Set preemptive_mode = true on the policy and push more parallel jobs than there are pause pods.
2. Check that pause_pods_reserved reaches the idle count, no_free_executor failures climb, job pods never outnumber pause pods, and the remaining jobs wait in the GitLab queue.
3. Send SIGQUIT to one replica with a job running and watch its pause pod Deployment go to zero within one reconcile.

What are the relevant issue numbers?

gitlab-com/gl-infra/production-engineering#29733