Intermittent error during zonal deployment in production
https://ops.gitlab.net/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/jobs/2076769
We saw a deploy failure on one of the zonal clusters
[tiller] 2020/10/13 14:05:53 warning: Upgrade "gitlab" failed: Get "https://34.73.149.139/api/v1/namespaces/gitlab/services/gitlab-gitlab-shell": dial tcp 34.73.149.139:443: connect: connection refused
[storage] 2020/10/13 14:05:53 updating release "gitlab.v1"
[storage/driver] 2020/10/13 14:05:53 update: failed to update: Put "https://34.73.149.139/api/v1/namespaces/kube-system/secrets/gitlab.v1": dial tcp 34.73.149.139:443: connect: connection refused
[tiller] 2020/10/13 14:05:53 warning: Failed to update release gitlab: Put "https://34.73.149.139/api/v1/namespaces/kube-system/secrets/gitlab.v1": dial tcp 34.73.149.139:443: connect: connection refused
It appears to be network related, but unsure.
At the very least I would like to make sure we are getting all the logs we need now from Kibana, related to https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/11452#note_428975878
this was resolved with a retry.