(Azure/AKS/ACI/Windows) kubernetes executor runner-helper cant be sheduled on ACI virtualnodes: a secret volume cant be mounted

Summary

We are using gitlab kubernetes executor to run build on windows nodes within an aks cluster. We tried to migrate to aci serverless containers, then the executor tries to schedule a pod an that virtual nodes but fails with:

api call to https://management.azure.com/subscriptions/xxxxxx/resourceGroups/xxxxxxxx/providers/Microsoft.ContainerInstance/containerGroups/runner-container-id?api-version=2018-10-01: got HTTP response status code 400 error code "SecretVolumesNotSupportedOnWindows": Secret volumes are not supported for Windows containers.

image

Steps to reproduce

Create an aks cluster, enable aci virtual nodes by using

Invoke-WebRequest -OutFile virtual-kubelet-latest.tgz https://github.com/virtual-kubelet/azure-aci/raw/master/charts/virtual-kubelet-latest.tgz;
helm install "virtual-kubelet" virtual-kubelet-latest.tgz --set provider=azure --set providers.azure.targetAKS=true --set providers.azure.masterUri=https://build-63765f9c.hcp.westeurope.azmk8s.io:443 --set nodeName=virtual-kubelet
helm install "virtual-kubelet-win" virtual-kubelet-latest.tgz --set provider=azure --set providers.azure.targetAKS=true --set providers.azure.masterUri=https://build-63765f9c.hcp.westeurope.azmk8s.io:443 --set nodeName="virtual-kubelet-win" --set "nodeOsType=Windows"
Remove-Item virtual-kubelet-latest.tgz

Install gitlab executor by helm and the start build

.gitlab-ci.yml 
# This image is the microsoft/framework/sdk:4.8-windwosservercore-1809 + out build tools + pwsh 7.1.5
image: hsbuildregistry.azurecr.io/hanseaticsoft/cfm/cfm-build-container:202111141452

stages:
  - build

build:
  tags:
    - kubernetes
    - windows
  stage: build
  script:
    - msbuild --version```
</details>

<details>
<summary> helmchartvalues.yml </summary>

```yml
## GitLab Runner Image
##
## By default it's using gitlab/gitlab-runner:alpine-v{VERSION}
## where {VERSION} is taken from Chart.yaml from appVersion field
##
## ref: https://hub.docker.com/r/gitlab/gitlab-runner/tags/
##
## Note: If you change the image to the ubuntu release
##       don't forget to change the securityContext;
##       these images run on different user IDs.
##
# image: gitlab/gitlab-runner:alpine-v11.6.0


imagePullPolicy: IfNotPresent

gitlabUrl: https://gitlab.com

runnerRegistrationToken: xxxxxx

terminationGracePeriodSeconds: 3600

concurrent: 2

checkInterval: 5

logLevel: debug

rbac:
  create: true
  
  rules:
    - resources: [ "pods", "secrets" ]
      verbs: [ "get", "list", "watch", "create", "patch", "delete" ]
    - apiGroups: [ "" ]
      resources: [ "pods/exec", "configmaps", "pods/attach", "secrets" ]
      verbs: [ "create", "patch", "delete", "update" ]
      
  clusterWideAccess: true
  
  podSecurityPolicy:
    enabled: false
    resourceNames:
    - gitlab-runner

metrics:
  enabled: true
  
runners:
  config: |
    [[runners]]
      shell = "pwsh"
      privileged = true
      executor = "kubernetes"
      environment = ["FF_USE_POWERSHELL_PATH_RESOLVER=true","FF_USE_WINDOWS_LEGACY_PROCESS_STRATEGY=false"]

      [runners.kubernetes]
        cpu_limit = "3"
      
      [runners.kubernetes.node_selector]
        "kubernetes.io/role" = "agent"
        "type" = "virtual-kubelet"
        "beta.kubernetes.io/os" = "windows"

      [runners.kubernetes.node_tolerations]
        "virtual-kubelet.io/provider=azure" = "NoSchedule"

  tags: "kubernetes, windows"

  name: "hs-build-cfm"
  
securityContext:
  runAsUser: 0
  fsGroup: 0
  
nodeSelector:
  "kubernetes.io/arch": "amd64"
  "kubernetes.io/os": "linux"

Actual behavior

Pods fail to schedule, due that they cant insert secrets in aci windows caontainers

Expected behavior

Pods start and executed job

Relevant logs and/or screenshots

image

Possible fixes

This seems to be an issue with aci and windows, somehow there seem to be workarounds like mopunting pvc instead of that secret

Edited by Johannes Rittmeier