The container name is already in use [Kubernetes Executor]
Summary
After moving to the group-wide K8s cluster we started experiencing failed builds. It happens several times a day and became a bit annoying.
In Gitlab it appears as an endless (till the timeout) train of the following message: Waiting for pod gitlab-managed-apps/runner-jqsuxeuc-project-35-concurrent-02sgpn to be running, status is Pending
Examining the pod's status we can see
Failed create pod sandbox: rpc error: code = Unknown desc = failed to create a sandbox for pod "runner-jqsuxeuc-project-35-concurrent-02sgpn": operation timeout: context deadline exceeded
message followed by
Failed create pod sandbox: rpc error: code = Unknown desc = failed to create a sandbox for pod "runner-jqsuxeuc-project-35-concurrent-02sgpn": Error response from daemon: Conflict. The container name "/k8s_POD_runner-jqsuxeuc-project-35-concurrent-02sgpn_gitlab-managed-apps_9158aaa1-ad9f-11e9-9b78-42010a84007f_0" is already in use by container "06d70c6ff225fd4e8798f58652c124ac832da0e5f7d2687b96eb9fd9014f5d91". You have to remove (or rename) that container to be able to reuse that name.
Steps to reproduce
It's quite difficult to reproduce since it happens sporadically for different jobs spread across different projects. It happens with different runners (we've got several instances of runners deployed, each tagged for use in particular jobs).
Actual behavior
Timeout and a failed build due to the troubles with a container.
Expected behavior
A runner pod up & running.
Relevant logs and/or screenshots
See the logs above.
Environment description
GKE
A dedicated node pool for runners with Container-Optimized OS
K8s version: 1.13.6-gke.13
Container runtime: docker://18.9.3
Used GitLab Runner version
Running with gitlab-runner 11.11.2 (ac2a293c)