Define PATH inside of helper image

Overview

Hello all,

We are running gitlab-runner on Kubernetes and our cloud provider recently upgraded the underlying containerd container runtime to v1.2.8.

There is an underlying change, which causes the following errors on the runners helper image:

failed to create containerd task: OCI runtime create failed: container_linux.go:345: starting container process caused "exec: \"sh\": executable file not found in $PATH": unknown

Steps to reproduce on kubernetes with containerd v1.2.8:

kubectl run -it --image gitlab/gitlab-runner-helper:x86_64-f100a208 msvechltest --command -- sh
kubectl run --generator=deployment/apps.v1 is DEPRECATED and will be removed in a future version. Use kubectl run --generator=run-pod/v1 or kubectl create instead.
Error from server (NotFound): pods "" not found
# we dont get a shell...

Further debugging shows:

kubectl describe pod msvechltest-5c77c798b7-nn2dp
Last State:     Terminated
     Reason:       StartError
     Message:      failed to create containerd task: OCI runtime create failed: container_linux.go:345: starting container process caused "exec: \"sh\": executable file not found in $PATH": unknown

There already is an issue open on containerd about this, and a MR is already scheduled for the next release which fixes this behaviour in some cases.

Apparently this happens on images where no default $PATH variable is set. While a fix for this is scheduled to be released soon, it might make sense to fix the environment of the helper image.

Proposal

The root-cause stems from https://gitlab.com/gitlab-org/gitlab-runner/blob/master/shells/bash.go#L227 where sh is executed without an absolute path. While it makes sense to have it without an absolute path here, adding a default $PATH to the helper image might make sense.