Kubernetes Windows runner pod fails init when mounting PVC to builds_dir
Summary
We (PTC Kepware) have Kubernetes runners in an AKS cluster. Our main repo is large, so we're mounting a PVC to the builds_dir so we can use a git fetch strategy, avoiding a ~5 minute repo pull time every job. This works fine for Linux, but not for Windows. When mounting the pre-existing clean PVC to the builds_dir location, the init container will fail with the error C:\builds\*: The system cannot find the file specified.
Steps to reproduce
- Create a storage class and PVC in an AKS cluster
- Use Helm to launch Windows Kubernetes runners in the same cluster. In the helm chart values file, set runners.builds_dir and runners.kubernetes.volumes.pvc.mount_path to "/builds"
- Run a job with that runner
.gitlab-ci.yml
test-job-win:
stage: test
tags:
- wink8s
script:
- echo 'test job'; sleep 300
rules:
- when: manual
Actual behavior
The pod spins up, binds the PVC successfully (and does not bind an empty_dir for the custom build_dir), pulls the helper image for the init container, then starts the init container. The init container then immediately fails, only providing the following log to stderr:
C:\builds\*: The system cannot find the file specified.
Expected behavior
The init container proceeds as normal, and the build container uses the bound and mounted PVC as the build directory.
Relevant logs and/or screenshots
job log
Running with gitlab-runner 15.8.0 (12335144)
on gitlab-runner-windows-c8fb5688f-s7nd5 -aEALdZs, system ID: r_JjbTzOPa7p3c
feature flags: FF_USE_POWERSHELL_PATH_RESOLVER:true
Resolving secrets
00:00
Preparing the "kubernetes" executor
00:00
Using Kubernetes namespace: gitlab-managed-apps
Using Kubernetes executor with image mcr.microsoft.com/dotnet/framework/sdk:4.8-windowsservercore-ltsc2019 ...
Using attach strategy to execute scripts...
Preparing environment
08:20
Waiting for pod gitlab-managed-apps/runner--aealdzs-project-876-concurrent-0mpw97 to be running, status is Pending
ERROR: Job failed (system failure): prepare environment: waiting for pod running: pod status is failed. Check https://docs.gitlab.com/runner/shells/index.html#shell-profile-loading for more information
Environment description
Runners are launched via a helm chart to a private AKS cluster. To accommodate the ReadWriteOnce nature of the AzureDisk persistent volumes, the target node pool is a singular node. Linux runners with the same configuration and a separate PVC from the same storage class work without issue.
config in values.yaml
[[runners]]
environment = ["FF_USE_POWERSHELL_PATH_RESOLVER=1"]
executor = "kubernetes"
builds_dir = "/builds"
[runners.kubernetes]
namespace = "{{.Release.Namespace}}"
image = "mcr.microsoft.com/dotnet/framework/sdk:4.8-windowsservercore-ltsc2019"
helper_image = "gitlab/gitlab-runner-helper:x86_64-v14.10.2-servercore1809"
privileged = true
poll_timeout = 30000
poll_interval = 500
[[runners.kubernetes.volumes.pvc]]
name = "runnerstoragewin"
mount_path = "/builds"
[runners.kubernetes.pod_labels]
"gitlab-runner-os" = "windows"
[runners.kubernetes.node_selector]
"kubernetes.io/arch" = "amd64"
"kubernetes.io/os" = "windows"
"node.kubernetes.io/windows-build" = "10.0.17763"
"application" = "gitlabrunner"
[runners.feature_flags]
FF_GITLAB_REGISTRY_HELPER_IMAGE = true
storage class spec
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: runnerstorage
provisioner: disk.csi.azure.com
parameters:
skuName: StandardSSD_LRS
reclaimPolicy: Retain
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
pvc spec
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: runnerstoragewin
namespace: gitlab-managed-apps
spec:
storageClassName: runnerstorage
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 50Gi
Used GitLab Runner version
Running with gitlab-runner 15.8.0 (12335144)
on gitlab-runner-windows-c8fb5688f-s7nd5 -aEALdZs, system ID: r_JjbTzOPa7p3c
feature flags: FF_USE_POWERSHELL_PATH_RESOLVER:true