Allow GitLab Runner to process named PodSpec(s) set in a .gitlab.ci yml pipeline file
## Overview Enabling support for named GitLab Runner PodSpec(s) to configure the Runner worker will unlock value for users of GitLab for [scientific workflows](https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/3114#note_1302779385). For example, data scientists, machine learning researchers, or engineers will require GPU-enabled computing on public cloud Kubernetes infrastructure. To use GPU-enabled compute means creating different podspecs for each class of GPU-enabled compute offered. Today, the podspec can only be set in the runner config.toml file. So first - this means the administrator of the self-managed Runner has to configure the podspec for each compute type. Secondly, every job on the specified Runner will request GPU-enabled resources regardless of whether the job requires a GPU. As a workaround, customers can configure multiple runners, each with its own pod spec. That approach is not efficient or flexible. ## Value proposition (customer verbatims) - "Here is one specific example: say we added a [new type of node](https://kubernetes.io/docs/tasks/manage-gpus/scheduling-gpus/#clusters-containing-different-types-of-gpus) to a Kubernetes cluster, and we wanted to use a node selector to run a pod on that specific node. It would be easier for the CI pipeline developers to make a code change (`.gitlab-ci.yml`) than to have to contact a system administrator to modify the Runner's configuration (`config.toml`) to add a new named PodSpec. - The workaround of `multiple names pod specs` is not ideal as "there are common situations that have a number of permutations. Take, for example, the GPU specification for GKE Autopilot workloads as shown in the [Google documentation](https://cloud.google.com/kubernetes-engine/docs/how-to/autopilot-gpus#request-gpus). If we wanted to cover all GPU type/quantity options, that would be eight unique pod specs. Not an intractable number, but slightly cumbersome." ## Proposal Allow users to configure a podspec in the `.gitlab-ci.yml` pipeline. ## Example ### config.toml ```toml ... [[runners]] [runners.kubernetes] [[runners.kubernetes.pod_spec]] name = "spec-a100-3" patch = ''' nodeSelector: cloud.google.com/gke-accelerator: "nvidia-tesla-a100" containers: - name: "build" resources: limits: nvidia.com/gpu: 2 ''' patch_type = "strategic" ... ``` ### gitlab-ci ```yaml pod_specs: - name: spec-a100-1 patch_type: "strategic" patch: | hostname: "my_host" - name: spec-a100-2 patch_type: "json" patch: | [{"op": "replace", "path": "/terminationGracePeriodSeconds", "value": 60}] job01: pod_specs: - spec-a100-2 script: - ./cmd01 - ./cmd02 job02: pod_specs: - spec-a100-1 - spec-a100-3 script: - ./cmd01 - ./cmd02 job03: script: - ./cmd01 - ./cmd02 services: - name: my-postgres:11.7 alias: db-postgres entrypoint: ["/usr/local/bin/db-postgres"] command: ["start"] - pod_specs: - spec-a100-1 - spec-a100-3 ```
issue