When a deployment to Kubernetes fails, the master branch is now inconsistent with what has been deployed
Our deployment procedure for Kubernetes assumes that deployments would always succeed. In times for which they do not, helm will automatically roll back the desired change. This presents an issue as the master branch is no longer an accurrate representation for what is running on the cluster. Utilize this issue to track what we can do to ensure that master is a reflection of what exists on a cluster.
Thoughts
- We would want to setup a failure job that performs work, potentially reverting the change
- Issues with linked MR's would need some form of communication to know that a deploy was unsuccessful, anything that may have been auto closed may need to be reopened