Running git lab runner in AKS virtual node Failing

Gitlab (kubernetes) runner pods failing to run on AKS virtual nodes ( but it is fine on normal AKS nodes). The runner master is running in normal node. When I biuild a job , the runner is not coming up and the runner master shows the following error. All the details are provided below.

  1. using helm Install, installed the gitlab runners with following values.yaml ref : https://docs.gitlab.com/runner/install/kubernetes.html#installing-gitlab-runner-using-the-helm-chart

gitlabUrl: https://gitxxx.xxxxx.net/

runnerRegistrationToken: "xXXx"

concurrent: 25

checkInterval: 30

logLevel: debug logFormat: text preEntrypointScript: | echo "hello Siju sAsmuel"

rbac: create: true resources: ["pods", "pods/exec", "secrets", "namespaces"] verbs: ["get", "list", "watch", "create", "patch", "delete"] clusterWideAccess: true runners: image: ubuntu:16.04 tags: "virt-node-new1" privileged: true imagePullPolicy: "always" imagePullSecrets: [regcred] builds: cpuRequests: 200m memoryRequests: 256Mi services: cpuRequests: 200m memoryRequests: 256Mi helpers: cpuRequests: 200m memoryRequests: 256M nodeSelector: kubernetes.io/role: agent beta.kubernetes.io/os: linux type: virtual-kubelet nodeTolerations: - key: virtual-kubelet.io/provider operator: Exists pollTimeout: 980

Which installed the runner master properly in AKS cluster . For the runner to schedule to virtual nodes added the nodeselctor and nodeTolerations

  1. Simple .gitlan-ci.yaml
stages:
    - test

before_script:

test_runner:
  stage: test
  tags:
    - virt-node-new1
  image:
    name: ubuntu:16.04
    entrypoint: [""]
  cache:
    paths:
      - "/runner-cache"
  script:
    - ls
    - echo "Test job running (3 minute minute sleep) . . ."
    - sleep 1m
    - pwd
    - echo "path above . . ."
    - sleep 1m
    - echo "Test done."
  only:
    - virtualnodetest
  1. Error in the runner master log : time="2020-09-24T22:00:25Z" level=debug msg="Executing build stage" build_stage=prepare_script job=300950 project=466 runner=ExSz47Ux time="2020-09-24T22:00:25Z" level=debug msg="\x1b[36;1mPreparing environment\x1b[0;m" job=300950 project=466 runner=ExSz47Ux time="2020-09-24T22:00:25Z" level=debug msg="Starting Kubernetes command..." job=300950 project=466 runner=ExSz47Ux time="2020-09-24T22:00:25Z" level=debug msg="Setting up secrets" job=300950 project=466 runner=ExSz47Ux time="2020-09-24T22:00:25Z" level=debug msg="Checking for jobs... nothing" runner=ExSz47Ux time="2020-09-24T22:00:25Z" level=debug msg="Setting up build pod" job=300950 project=466 runner=ExSz47Ux time="2020-09-24T22:00:25Z" level=debug msg="Creating build pod" job=300950 project=466 runner=ExSz47Ux time="2020-09-24T22:00:25Z" level=debug msg="Creating pod proxy services" job=300950 project=466 runner=ExSz47Ux time="2020-09-24T22:00:25Z" level=debug msg="Starting in container "helper" the command ["gitlab-runner-build"] with script: #!/usr/bin/env bash\n\nset -eo pipefail\nset +o noclobber\n: | eval $'echo "Running on $(hostname) via gitlab-runner-virt-node-new1-gitlab-runner-5fd4b9677c-mpm7s..."\n'\nexit 0\n" job=300950 project=466 runner=ExSz47Ux time="2020-09-24T22:00:28Z" level=debug msg="Appending trace to coordinator... ok" code=202 job=300950 job-log=0-624 job-status=running runner=ExSz47Ux sent-log=0-623 status="202 Accepted" update-interval=30s time="2020-09-24T22:00:28Z" level=debug msg="Container "helper" exited with error: pod status is failed" job=300950 project=466 runner=ExSz47Ux time="2020-09-24T22:00:28Z" level=error msg="Job failed (system failure): prepare environment: pod status is failed. Check https://docs.gitlab.com/runner/shells/index.html#shell-profile-loading for more information" duration=3.099527424s job=300950 project=466 runner=ExSz47Ux time="2020-09-24T22:00:28Z" level=debug msg="Appending trace to coordinator... ok" code=202 job=300950 job-log=0-855 job-status=running runner=ExSz47Ux sent-log=624-854 status="202 Accepted" update-interval=30s time="2020-09-24T22:00:28Z" level=debug msg="Submitting job to coordinator... ok" code=200 job=300950 job-status= runner=ExSz47Ux time="2020-09-24T22:00:28Z" level=warning msg="Failed to process runner" builds=0 error="prepare environment: pod status is failed. Check https://docs.gitlab.com/runner/shells/index.html#shell-profile-loading for more information" executor=kubernetes runner=ExSz47Ux

Please suggest any resolution ? Is it supposed to run on virtual node? No gitlab runner pods are being spun up to check additional erorrs

thx

Edited by Si Channel