Create a runbook to manually change clusters
That being said, I do think our ability to move away from "pet" clusters to "cattle" clusters in some capacity would be ideal. An example would be the GKE 1.16 upgrade I am currently doing. When I look at the change request, I see that realistically I have no rollback steps, as GKE is outside my control (as in, I send the API request to Google to upgrade, then cross my fingers). Having a runbook for standing up a new cluster quickly, pointing k8s-workloads/gitlab-com and k8s-workloads/gitlab-helmfiles to that, and ensuring we could get something back up (then update haproxy) would actually be quite useful. Writing and testing this runbook could be a good first step. Eventually moving to spinning up new clusters every couple of months just for good practice could be a longer term goal.
I think this is an excellent thing to strive for, even in a short term before we start migrating some more critical sync workloads. I appreciate the fact that this might not be easy to do now, but I am in agreement with Graeme that we want to have at least some level of practice to allow us to react to unexpected failures with our provider.