Support Node affinity for pause pods with Kubernetes executor
Summary
I wanted to try pause pods to speed up jobs startup. However the pause pods don't spawn on the right nodes. They seem to ignore the runners.kubernetes.affinity.node_affinity configuration.
Steps to reproduce
- Configure a Kubernetes runner with
runners.kubernetes.affinity.node_affinity - Enable pause pods
Actual behavior
The pause pods deployment ignores the node affinity
Expected behavior
The pause pods should deploy on the same nodes as the jobs will
Relevant logs and/or screenshots
In the following manifest, we can see the configured node toleration is here, but not the node affinity :
runner-pause deployment YAML
apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
deployment.kubernetes.io/revision: '1'
runner.gitlab.com/pause-heartbeat: '2026-07-21T09:05:43Z'
creationTimestamp: '2026-07-21T08:52:53Z'
generation: 12
labels:
manager.runner.gitlab.com/id-short: kUpb21O-A
manager.runner.gitlab.com/system-id: r_oIQUNuA5lMFH
runner.gitlab.com/pause-pod: 'true'
name: runner-pause-kupb21o-a-roiqunua5lmfh
namespace: gitlab
resourceVersion: '1784624743122959000'
uid: 827b7e7d-5124-45de-a558-d558cd2b6c4c
spec:
progressDeadlineSeconds: 600
replicas: 5
revisionHistoryLimit: 10
selector:
matchLabels:
manager.runner.gitlab.com/id-short: kUpb21O-A
manager.runner.gitlab.com/system-id: r_oIQUNuA5lMFH
runner.gitlab.com/pause-pod: 'true'
strategy:
rollingUpdate:
maxSurge: 25%
maxUnavailable: 25%
type: RollingUpdate
template:
metadata:
labels:
manager.runner.gitlab.com/id-short: kUpb21O-A
manager.runner.gitlab.com/system-id: r_oIQUNuA5lMFH
runner.gitlab.com/pause-pod: 'true'
spec:
containers:
- image: registry.k8s.io/pause:3.10
imagePullPolicy: IfNotPresent
name: pause
resources:
requests:
cpu: 500m
memory: 1Gi
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
dnsPolicy: ClusterFirst
priorityClassName: pause-pods-priority-class
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
serviceAccount: default
serviceAccountName: default
terminationGracePeriodSeconds: 0
tolerations:
- effect: NoSchedule
key: preemptible
operator: Equal
value: 'true'Environment description
I am using Kubernetes executor on GCP with a self-hosted GitLab instance. We use node affinity in order for jobs to prefer less expensive spot nodes if available.
config.toml contents
[runners.kubernetes.autoscaler]
max_pause_pods = 10
pause_pod_priority_class_name = "pause-pods-priority-class"
[[runners.kubernetes.autoscaler.policy]]
idle_count = 0
periods = ["* * * * *"]
[[runners.kubernetes.autoscaler.policy]]
idle_count = 5
periods = ["* 7-20 * * mon-fri"]
[runners.kubernetes.node_tolerations]
"preemptible=true" = "NoSchedule"
# Set preference for spot VM but will fallback to non-spot if no spot VMs are available
[runners.kubernetes.affinity.node_affinity]
[[runners.kubernetes.affinity.node_affinity.preferred_during_scheduling_ignored_during_execution]]
weight = 100
[runners.kubernetes.affinity.node_affinity.preferred_during_scheduling_ignored_during_execution.preference]
[[runners.kubernetes.affinity.node_affinity.preferred_during_scheduling_ignored_during_execution.preference.match_expressions]]
key = "cloud.google.com/gke-spot"
operator = "In"
values = ["true"]
[runners.kubernetes.affinity.node_affinity.required_during_scheduling_ignored_during_execution]
[[runners.kubernetes.affinity.node_affinity.required_during_scheduling_ignored_during_execution.node_selector_terms]]
[[runners.kubernetes.affinity.node_affinity.required_during_scheduling_ignored_during_execution.node_selector_terms.match_expressions]]
key = "ci"
operator = "In"
values = ["true"]Used GitLab Runner version
Docker image : gitlab/gitlab-runner:v19.2.0
Possible fixes
I didn't find anything related to node affinity in the pause_pod_manager
I guess I could mitigate the issue by using node selectors / tolerations, but then having a preference for spot nodes is impossible