Auto DevOps : Helm cannot upgrade over initial failed release
## Steps to reproduce 1. Have a failed Helm release (e.g. forgot to set base domain) 2. Fix issue from 1, run Auto DevOps pipeline again 3. Failure below occurs ``` $ deploy secret "review-secrets-ayz3io-secret" deleted secret "review-secrets-ayz3io-secret" replaced Initializing... Error: UPGRADE FAILED: "review-secrets-ayz3io" has no deployed releases ERROR: Job failed: exit code 1 ``` ## Notes * This seems to only occur if **ALL** releases for that chart have only ever failed. * If there is a successful release in the past, then Helm can upgrade over a previously failed release ## Workaround As shown by @tnir in https://gitlab.com/gitlab-org/gitlab-ce/issues/54760#note_146627376 1. Run the following script locally. For the `production` job, the `<chart-name>` is `production`. For other jobs, run `helm ls` which should show you which chart to delete. ``` $ tiller -version v2.13.0 $ export KUBE_NAMESPACE=<CI_PROJECT_NAME>-<CI_PROJECT_ID>-<CI_PROJECT_ENVIRONMENT> $ export TILLER_NAMESPACE=$KUBE_NAMESPACE $ tiller -listen localhost:44134 & [1] 28659 $ export HELM_HOST="localhost:44134" $ helm init --client-only $ helm delete <release-name> --purge --tiller-namespace $KUBE_NAMESPACE release "<release-name>" deleted ``` 2. Retry the deploy job ## Links See also discussion in https://github.com/helm/helm/issues/3208 and https://github.com/helm/helm/issues/3353 ## Proposed fix Invoke `--atomic` by default when installing releases using Auto DevOps (this can be added in https://gitlab.com/gitlab-org/cluster-integration/auto-deploy-image) Add support to disable calling `--atomic` by checking `AUTO_DEVOPS_ATOMIC_RELEASE` env variable (Why? In case where users may attach a PVC that is deleted on reclaim, we want to give users the option to opt out on this)
issue