Skip to content

GitLab runner upgrade connection cannot find helper

Summary

Kubernetes executor cannot upgrade connection. That error message isn't very descriptive. Is GitLab Kuberenetes runner not compatible with containerd?

Running with gitlab-runner 12.3.0 (a8a019e0)
  on gitlab-runner2-gitlab-runner-589c475db9-cb7bh K7Pqz_rm
Using Kubernetes namespace: default
Using Kubernetes executor with image php:5.6 ...
Waiting for pod default/runner-k7pqzrm-project-2-concurrent-026g6n to be running, status is Pending
Waiting for pod default/runner-k7pqzrm-project-2-concurrent-026g6n to be running, status is Pending
ERROR: Job failed (system failure): unable to upgrade connection: container not found ("helper")

Steps to reproduce

  1. Install GitLab on prem with helm chart on k3s. (values from values-minikube.yaml example with global.edition: ce and global.ingress.tls.enabled: false and runner enabled.
  2. Attempt to run any pipeline. i.e. php auto devops example.
  3. Observe failing jobs.

Actual behavior

Job fails.

Relevant logs and/or screenshots

kubectl get events:

Normal  Pulled         pod/runner-k7pqzrm-project-2-concurrent-026g6n    Container image "gitlab/gitlab-runner-helper:x86_64-a8a019e0" already present on machine
Normal  Created        pod/runner-k7pqzrm-project-2-concurrent-026g6n    Created container helper
14m         Warning   Failed                         pod/runner-k7pqzrm-project-2-concurrent-026g6n    Error: failed to create containerd task: OCI runtime create failed: container_linux.go:337: starting container process caused "exec: \"sh\": executable file not found in $PATH": unknown
Normal  Pulling        pod/runner-k7pqzrm-project-2-concurrent-026g6n    Pulling image "mysql"
Normal  Pulled         pod/runner-k7pqzrm-project-2-concurrent-026g6n    Successfully pulled image "mysql"
Normal  Created        pod/runner-k7pqzrm-project-2-concurrent-026g6n    Created container svc-0
Normal  Started        pod/runner-k7pqzrm-project-2-concurrent-026g6n    Started container svc-0
Normal  Killing        pod/runner-k7pqzrm-project-2-concurrent-026g6n    Stopping container build
Normal  Killing        pod/runner-k7pqzrm-project-2-concurrent-026g6n    Stopping container svc-0
Warning FailedKillPod  pod/runner-k7pqzrm-project-2-concurrent-026g6n    error killing pod: failed to "KillPodSandbox" for "25793f39-02de-47af-92ff-3fd2358aa495" with KillPodSandboxError: "rpc error: code = Unknown desc = failed to destroy network for sandbox \"4f1fda6c252447e6760f463342dcea8b92fc89e2abd4b8a7db24709a6248ad4c\": could not teardown ipv4 dnat: running [/var/lib/rancher/k3s/data/17a94af11136a3c0b873511b46c76bdeb66e47514fe377828e871ec242d1b3a0/bin/iptables -t nat -X CNI-DN-5fafd292f895661f8bc25 --wait]: exit status 1: iptables: No chain/target/match by that name.\n"