GitLab runner upgrade connection cannot find helper

Summary

Kubernetes executor cannot upgrade connection. That error message isn't very descriptive. Is GitLab Kuberenetes runner not compatible with containerd?

Running with gitlab-runner 12.3.0 (a8a019e0)
  on gitlab-runner2-gitlab-runner-589c475db9-cb7bh K7Pqz_rm
Using Kubernetes namespace: default
Using Kubernetes executor with image php:5.6 ...
Waiting for pod default/runner-k7pqzrm-project-2-concurrent-026g6n to be running, status is Pending
Waiting for pod default/runner-k7pqzrm-project-2-concurrent-026g6n to be running, status is Pending
ERROR: Job failed (system failure): unable to upgrade connection: container not found ("helper")

Steps to reproduce

  1. Install GitLab on prem with helm chart on k3s. (values from values-minikube.yaml example with global.edition: ce and global.ingress.tls.enabled: false and runner enabled.
  2. Attempt to run any pipeline. i.e. php auto devops example.
  3. Observe failing jobs.

Actual behavior

Job fails.

Relevant logs and/or screenshots

kubectl get events:

Normal  Pulled         pod/runner-k7pqzrm-project-2-concurrent-026g6n    Container image "gitlab/gitlab-runner-helper:x86_64-a8a019e0" already present on machine
Normal  Created        pod/runner-k7pqzrm-project-2-concurrent-026g6n    Created container helper
14m         Warning   Failed                         pod/runner-k7pqzrm-project-2-concurrent-026g6n    Error: failed to create containerd task: OCI runtime create failed: container_linux.go:337: starting container process caused "exec: \"sh\": executable file not found in $PATH": unknown
Normal  Pulling        pod/runner-k7pqzrm-project-2-concurrent-026g6n    Pulling image "mysql"
Normal  Pulled         pod/runner-k7pqzrm-project-2-concurrent-026g6n    Successfully pulled image "mysql"
Normal  Created        pod/runner-k7pqzrm-project-2-concurrent-026g6n    Created container svc-0
Normal  Started        pod/runner-k7pqzrm-project-2-concurrent-026g6n    Started container svc-0
Normal  Killing        pod/runner-k7pqzrm-project-2-concurrent-026g6n    Stopping container build
Normal  Killing        pod/runner-k7pqzrm-project-2-concurrent-026g6n    Stopping container svc-0
Warning FailedKillPod  pod/runner-k7pqzrm-project-2-concurrent-026g6n    error killing pod: failed to "KillPodSandbox" for "25793f39-02de-47af-92ff-3fd2358aa495" with KillPodSandboxError: "rpc error: code = Unknown desc = failed to destroy network for sandbox \"4f1fda6c252447e6760f463342dcea8b92fc89e2abd4b8a7db24709a6248ad4c\": could not teardown ipv4 dnat: running [/var/lib/rancher/k3s/data/17a94af11136a3c0b873511b46c76bdeb66e47514fe377828e871ec242d1b3a0/bin/iptables -t nat -X CNI-DN-5fafd292f895661f8bc25 --wait]: exit status 1: iptables: No chain/target/match by that name.\n"