Draft: Enable dynamic PersistentVolumeClaims in Kubernetes

What does this MR do?

Adds the capability to configure the Kubernetes executor to instantiate a VolumeClaimTemplate whenever it creates a pod. The template is instantiated by injecting CI variables into it. The MR also enables injecting variables to the name of a pvc volume attached to the pod.

By using these two changes together, you can reserve and mount volume claims for pods dynamically.

Why was this MR needed?

Currently the Kubernetes runner always clones the git repository from scratch; it has no cache. It would be possible to configure a cache into use manually by having a single huge volume, which is mounted in a multi-user mode and mounted to every pod. The concurrent pods could then use different folders via CI_CONCURRENT_ID. But multi-user volumes are slower and less safe than single-user ones.

With the changes in this MR, it is possible to configure a volume claim template, where the name includes CI_CONCURRENT_ID and mount it. This allows a per-concurrent-execution volume like in the docker executor. Under the hood these can be allocated from a fast per-node storage class.

What's the best way to test this MR?

1. Install this version of gitlab-runner on a Kubernetes cluster.
2. Create or select a storage class for the build volumes. Here is an example of what worked for me in Azure:

   kind: StorageClass
   apiVersion: storage.k8s.io/v1
   metadata:
     name: some-storage-class
   provisioner: disk.csi.azure.com
   parameters:
     skuname: StandardSSD_LRS
   allowVolumeExpansion: true
   # Keep the volumes around. They are used for git clones of build jobs
   # and we don't want to clone from scratch each time.
   # We may need some mechanism to ensure the system doesn't
   # leak these over time.
   reclaimPolicy: Retain
   # The volume is created when first used.
   volumeBindingMode: WaitForFirstConsumer

3. Configure a pvc mounting and volume template, both with the name set to ´pvc-$CI_CONCURRENT_ID´ and configure the build directory to reside on that volume.
4. Observe that repeated jobs from the same project with GIT_STRATEGY set to fetch will no longer have to do the clone from scratch every time.

[[runners]]
  executor = "kubernetes"
  builds_dir = "/mnt/dynamic-pvc"
  [runners.kubernetes]
    # Creates a PersistentVolumeClaim for each concurrent job execution
    [[runners.kubernetes.volume_claim_templates]]
      template = """
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc-$CI_CONCURRENT_ID
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 100Gi
  storageClassName: some-storage-class"""
    # Mounts the generated PersistentVolumeClaim to a folder.
    [[runners.kubernetes.volumes.pvc]]
      name = "pvc-$CI_CONCURRENT_ID"
      mount_path = "/mnt/dynamic-pvc"

What are the relevant issue numbers?

#27835

This MR continues on top of !4256 (merged), which enables the injection of variables to [[runners.kubernetes.volumes.pvc]].name.