On GitLab.com, the `Clusters::Cleanup::ServiceAccountWorker` sidekiq job has about a 12% failure rate

Worker Dashboard: https://dashboards.gitlab.net/d/sidekiq-worker-detail/sidekiq-worker-detail?var-worker=Clusters::Cleanup::ServiceAccountWorker

Failure Sidekiq Logs: Kibana Search

As measured over the past 3 day period on GitLab.com, the Clusters::Cleanup::ServiceAccountWorker sidekiq job has about a 12% failure rate.

Common errors include:

1. Kubeclient::HttpError: Timed out connecting to server
2. Gitlab::UrlBlocker::BlockedUrlError: Host cannot be resolved or invalid
3. NoMethodError: undefined method kubeclient for nil:NilClass

Proposal

Reduce failure fail to less than 10%.

Possibly validate kubernetes URL before persisting the job?

Verification

Is the SidekiqServiceWorkerExecutionErrorSLOViolation alert still firing?

https://alerts.gitlab.net/#/alerts?silenced=true&inhibited=false&active=true&filter=%7Balertname%3D%22SidekiqServiceWorkerExecutionErrorSLOViolation%22%2C%20env%3D%22gprd%22%2C%20worker%3D%22Clusters::Cleanup::ServiceAccountWorker%22%7D