Kubernetes executor: allow labels and annotations on pause pods (pause_pod_labels / pause_pod_annotations)

Description

The Kubernetes executor's pause pod feature ([runners.kubernetes.autoscaler], GitLab Runner 18.10+) builds the pause pod Deployment template with a fixed set of three labels and no annotations. In executors/kubernetes/autoscaler/pause_pod_manager.go (buildDeployment, v19.2.0) the pod template carries only:

labels := map[string]string{
    pausePodLabel: pausePodLabelValue,   // runner.gitlab.com/pause-pod=true
    runnerIDLabel: m.config.RunnerShortToken,
    systemIDLabel: m.config.SystemID,
}

and the PodSpec receives NodeSelector, Tolerations, ServiceAccountName, RuntimeClassName, ImagePullSecrets and PriorityClassName from the runner configuration. [runners.kubernetes.pod_labels] and [runners.kubernetes.pod_annotations] are applied to job pods but are not propagated to pause pods, and KubernetesAutoscalerConfig (common/config.go) exposes no equivalent setting for them.

Problem

Pause pods exist to hold capacity on the same nodes that job pods will use. In clusters where scheduling or policy depends on pod metadata, pause pods that carry none end up somewhere different from the job pods they are reserving capacity for, and the feature silently does nothing for that runner.

Concrete cases seen with a large enterprise customer running the Kubernetes executor on EKS with Karpenter (internal references below):

  1. Architecture placement via mutating admission webhook. The platform's webhook rewrites kubernetes.io/arch to amd64 unless a pod carries an opt-in annotation. Job pods carry it through pod_annotations and land on arm64 nodes as configured. Pause pods do not, so every pause pod for the three arm64 runner fleets was mutated to amd64, even though the runner had correctly copied kubernetes.io/arch: arm64 from node_selector into the template. Result: arm64 jobs were rejected with Preemption is not helpful for scheduling and waited for a new NodeClaim on every start, while the pause pods pinned amd64 capacity they could never hand over. The customer's only remediation was to set idle_count = 0 on those fleets, which disables the feature for the workloads that needed it most.
  2. Policy engines that require metadata. Cluster policies (Kyverno in this case) that require labels or annotations on every pod flag each pause pod as a violation. With several hundred pause pods this produced on the order of 24,000 PolicyViolation events within the event retention window, which is also a non-trivial write load on the API server.
  3. Cost attribution and observability. Team/cost-centre labels and monitoring tags (tags.datadoghq.com/*, Kubecost labels, and so on) are applied to job pods through pod_labels; pause pods, which are the idle cost, are the one workload that cannot be tagged.

Why blind propagation is not the right fix

Simply copying pod_labels / pod_annotations onto pause pods would be wrong in at least two common configurations:

  • karpenter.sh/do-not-disrupt: "true" is routinely set on job pods so a running job is never evicted. On pause pods it would make every warm node undisruptable and defeat consolidation.
  • pod-cleanup.gitlab.com/ttl (used with gitlab-runner-pod-cleanup) on pause pods would cause the cleanup tool to delete the warm pool on the configured interval.
  • pod_labels values may reference job variables ($CI_PROJECT_ID, and so on), which have no meaning for a pause pod.

Pause pods therefore need their own, explicit metadata settings rather than inheriting the job pod's.

Proposal

Add two optional maps to [runners.kubernetes.autoscaler], applied to the pause pod Deployment's pod template:

[runners.kubernetes.autoscaler]
  max_pause_pods = 30
  pause_pod_priority_class_name = "pause-pods"

  [runners.kubernetes.autoscaler.pause_pod_labels]
    "team" = "platform-ci"
    "tags.datadoghq.com/env" = "prd"

  [runners.kubernetes.autoscaler.pause_pod_annotations]
    "arm/include" = "true"

Implementation sketch, following the existing pattern for NodeSelector / Tolerations / ImagePullSecrets:

  1. common/config.go, KubernetesAutoscalerConfig: 
    PausePodLabels      map[string]string `toml:"pause_pod_labels,omitempty" json:"pause_pod_labels,omitempty" description:"Additional labels for pause pods"`
PausePodAnnotations map[string]string `toml:"pause_pod_annotations,omitempty" json:"pause_pod_annotations,omitempty" description:"Annotations for pause pods"`
  2. executors/kubernetes/autoscaler/provider.go, createManager: pass both maps into PausePodManagerConfig.
  3. executors/kubernetes/autoscaler/pause_pod_manager.go, buildDeployment: merge PausePodLabels into the pod template labels after the three runner-managed labels, ignoring (and logging at debug) any key in the runner.gitlab.com or manager.runner.gitlab.com namespaces so the Deployment selector cannot be broken, consistent with the existing rule for job pods that labels in the runner.gitlab.com namespace cannot be overridden. Set PausePodAnnotations as the pod template annotations. Leave Spec.Selector.MatchLabels as the three runner labels only.
  4. Documentation: add both settings to the "Autoscaler settings" table in docs/executors/kubernetes/_index.md, with a note that pod_labels / pod_annotations are intentionally not inherited and why (karpenter.sh/do-not-disrupt, pod-cleanup TTL).

Acceptance criteria

  • Pause pods created by a runner with pause_pod_labels / pause_pod_annotations set carry exactly those labels and annotations in addition to the three runner-managed labels.
  • Keys in the runner.gitlab.com and manager.runner.gitlab.com label namespaces are ignored.
  • Changing either map takes effect on the next reconcile without the runner needing to be restarted, or the behaviour on change is documented (today a config change already recreates the pool via ensureManager).
  • Existing configurations without the new keys are unchanged.
  • Unit tests in pause_pod_manager_test.go for merge, reserved-key filtering and the empty case.

Workaround until this ships

Pause pods already carry the label runner.gitlab.com/pause-pod=true. Admission policies that need to treat pause pods differently can match on that label today (for example, exempt them from an architecture rewrite, or from a required-annotations policy). This does not help the cost-attribution case.

Related

  • #39615 — pause pods ignore node_affinity. Same root cause class (pause pod template does not carry the job pod's scheduling constraints). If the fix for #39615 introduces a general [runners.kubernetes.autoscaler.pause_pod] template sub-table, the two settings proposed here belong in it.
  • #39395 — imagePullSecrets for pause pods (shipped in 19.1 as pause_pod_image_pull_secrets); this proposal follows the same shape.

Links to related issues and merge requests / references