Builds are stuck in "preparing"

Multiple users are observing that builds are stuck in preparing:

https://forum.gitlab.com/t/pipeline-stuck-in-preparing-on-kubernetes-deployment/25489

Impact

Only impacts projects which has a Kubernetes cluster (project or group level) and GitLab errored while creating the project namespace + project service account for the cluster.

Possible Cause

Looks like a regression from https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/25586

• @tkuah started a discussion:

  This call needs to be rescued?

  See https://sentry.gitlab.net/gitlab/gitlabcom/?query=is%3Aunresolved+BuildPrepareWorker for a list of error types - mostly Kubeclient::HttpError, OpenSSL::X509::CertificateError, but also other network related errors too.

  There's also StateMachines::InvalidTransition - Cannot transition status via :drop from :canceled => https://sentry.gitlab.net/gitlab/gitlabcom/issues/714313/?query=is:unresolved%20BuildPrepareWorker

  cc @tigerwnz