GitLab runner upgrade connection cannot find helper
Summary
Kubernetes executor cannot upgrade connection. That error message isn't very descriptive. Is GitLab Kuberenetes runner not compatible with containerd?
Running with gitlab-runner 12.3.0 (a8a019e0)
on gitlab-runner2-gitlab-runner-589c475db9-cb7bh K7Pqz_rm
Using Kubernetes namespace: default
Using Kubernetes executor with image php:5.6 ...
Waiting for pod default/runner-k7pqzrm-project-2-concurrent-026g6n to be running, status is Pending
Waiting for pod default/runner-k7pqzrm-project-2-concurrent-026g6n to be running, status is Pending
ERROR: Job failed (system failure): unable to upgrade connection: container not found ("helper")Steps to reproduce
- Install GitLab on prem with helm chart on k3s. (values from values-minikube.yaml example with global.edition: ce and global.ingress.tls.enabled: false and runner enabled.
- Attempt to run any pipeline. i.e. php auto devops example.
- Observe failing jobs.
Actual behavior
Job fails.
Relevant logs and/or screenshots
kubectl get events:
Normal Pulled pod/runner-k7pqzrm-project-2-concurrent-026g6n Container image "gitlab/gitlab-runner-helper:x86_64-a8a019e0" already present on machine
Normal Created pod/runner-k7pqzrm-project-2-concurrent-026g6n Created container helper
14m Warning Failed pod/runner-k7pqzrm-project-2-concurrent-026g6n Error: failed to create containerd task: OCI runtime create failed: container_linux.go:337: starting container process caused "exec: \"sh\": executable file not found in $PATH": unknown
Normal Pulling pod/runner-k7pqzrm-project-2-concurrent-026g6n Pulling image "mysql"
Normal Pulled pod/runner-k7pqzrm-project-2-concurrent-026g6n Successfully pulled image "mysql"
Normal Created pod/runner-k7pqzrm-project-2-concurrent-026g6n Created container svc-0
Normal Started pod/runner-k7pqzrm-project-2-concurrent-026g6n Started container svc-0
Normal Killing pod/runner-k7pqzrm-project-2-concurrent-026g6n Stopping container build
Normal Killing pod/runner-k7pqzrm-project-2-concurrent-026g6n Stopping container svc-0
Warning FailedKillPod pod/runner-k7pqzrm-project-2-concurrent-026g6n error killing pod: failed to "KillPodSandbox" for "25793f39-02de-47af-92ff-3fd2358aa495" with KillPodSandboxError: "rpc error: code = Unknown desc = failed to destroy network for sandbox \"4f1fda6c252447e6760f463342dcea8b92fc89e2abd4b8a7db24709a6248ad4c\": could not teardown ipv4 dnat: running [/var/lib/rancher/k3s/data/17a94af11136a3c0b873511b46c76bdeb66e47514fe377828e871ec242d1b3a0/bin/iptables -t nat -X CNI-DN-5fafd292f895661f8bc25 --wait]: exit status 1: iptables: No chain/target/match by that name.\n"