Schedule deployment migrations as a job on k8s
Note
Promoted to the epic &1556.
Problem Statement
Currently we host a set of deploy nodes, single VM's that are chef managed that are occasionally running low on disk and sit idle for the vast majority of their lifetime. They are responsible for running migrations at an appropriate time dictated by the Coordinator and Post Deploy Migration Pipelines. We should consider removing these one-off VM's in favor of running the Migrations inside of Kubernetes. We disable migrations currently to enable us to carefully manage and orchestrate a deployment. We should work up what it'll take to migrate towards Kubernetes for this work.
Follow up of gitlab-org/release-tools!1747 (comment 913775954)
My personal preference will be to schedule post-deployment migrations as a job on k8s ( @ggillies what's your opinion on this? ) but I think that adding the locking check on the new scheduled post-deployment pipeline (in deployer) is much less work at the cost of blocking new deployments during post-deployment migrations (which is not different from what we are doing today), so probably a better first iteration.
From: gitlab-org/release-tools!1747 (comment 915996170)
This would be my preference as well. While it might be a little bit trickier for us to implement, it has the following benefits
It would lay the groundwork for our release-tooling to introspect/manipulate and manage Kubernetes objects directly. This is likely something we are going to need the capacity to do for different things moving forward
It would allow us to move more work away from VMs (especially those that sit idle most of the time)
Our monitoring story can become stronger as we can leverage the Generic Kubernetes monitoring we already have (e.g. crashlooping pods, image pull errors, etc)
We can leverage Kubernetes to give us a longer history about all the post migration jobs and their output. As jobs are stored indefinitely (depending on implementation these can be automatically pruned).