Add support for Kubernetes PodSpec primitives: Pod Affinity and Pod Anti Affinity
Release notes
The Kubernetes Pod affinity feature enables you to constrain pods to specific nodes. As of 14.3, the GitLab Runner Kubernetes executor allows you to assign pods to nodes using the Kubernetes affinity type inter-pod affinity/anti-affinity.
Overview
Different CI jobs may need to leverage different type of hardware capabilities (eg. GPUs, specific CPUs)
A kubernetes cluster may consist of different CPU-gens and CPU types and you usually want your CI jobs to finish asap.
You usually want your jobs to prioritize for example nodes that have specific CPUs, but you don't want to waste all the other CPUs that are available, so you want your jobs to spill over to slower CPUs when running at capacity on the fast-class CPUs.
Example environment:
3 x nodes with Server grade CPUs (e.g. 2x Intel Xeon Scalable silver 4116) 12c (24c/48t total), 2.1/3.0 GHz, 512GB memory
3 x nodes with Consumer grade CPUs (e.g. 1x AMD Ryzen 7 2700X) 8c/16t, 3.7/4.3GHz, 64GB memory
Parallel workloads and are able to really benefit form the many cores available in the Xeon CPUs will complete faster on those, but many CI workloads benefit from the faster single-thread performance of the AMD Ryzen CPUs.
Proposal
Add support for adding nodeAffinity-objects for the Kubernetes Executor. in order for this to be a very useful feature, i think this needs to be done on 2 levels:
Support nodeAffinity in the executor configuration:
The two nodeAffinity primitives that should be supported:
requiredDuringSchedulingIgnoredDuringExecutionpreferredDuringSchedulingIgnoredDuringExecution
Since these are somewhat complex objects, one way to handle this might be to just inline json in the executor configuration, and pass that directly to the kubernetes client api by using *api.NodeAffinity toml:node_affinity -> where the content of node_affinity is a json blob (since Kubernetes does not support toml out of the box).
another way would be to directly parse and satisfy the api.NodeAffinity interface, this will however cause more maintenance as the kubernetes apis evolve.
A third way, is supporting a seperate podspec.yaml that gets loaded as a extra configuration file, this might require more changes in the executor configuration and support the overrides from the runner-config.toml in order to not introduce a breaking change and still support all the current usecases (other simpler objects could also easly be expressed as toml with fewer changes).
I'm not sure if the PodAffinity/PodAntiAffinity primitives makes sense at this point, but there might be use for them as well.
I'm willing to put in the effort to do this, but before I get to deep into these changes there's some discussions around how the implementation should be done, in order for these changes to get merged and for the broader community to be able to benefit from these changes.
Next steps / future evolution
At a later stage, it would be beneficial to support these types of options as job-options. I would guess this is what might be considered a special use-case, but it gives a lot of flexibility with the use of Kubernetes primitives
Example job
test job:
stage: initial
image: alpine:latest
script:
- echo "Hello Kubernetes"
tags:
- kubernetes
kubernetes:
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/e2e-az-name
operator: In
values:
- e2e-az1
- e2e-az2
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 1
preference:
matchExpressions:
- key: another-node-label-key
operator: In
values:
- another-node-label-value