Convert the blackout window to a read-only period
In discussion with @sytses on Friday's call, he noted that making GitLab.com read-only during the migration is far superior to a complete blackout, which is what we're doing at present (by disabling SSH and redirecting all HTTP+HTTPS requests to the blog post in HAProxy).
Which got me thinking: we don't need to blackout GitLab.com for the duration, we just need to ensure that updates cannot occur.
We have rudimentary support for a read-only/maintenance mode in-application, but I'd be loathe to rely on it without more development work, and I'm loathe to rely on more development work for the GCP Migration.
Instead, we could relax the HAProxy rules we're using so that HTTP GET+HEAD requests are permitted, while all others are rejected. Instead of redirecting people to the blog post, we'd put a link to the blog post in the broadcast message. This block would be more general than the in-application one - for instance, login and git clone may be broken by it - but the API would still be available and users would be able to browse their issues, etc.
A bit of thinking of the "Promote the secondary" phase may need to be done to maintain read-only availability.
At present, we:
- Update the DNS
- Promote postgresql in GCP to the primary
- Turn off postgresql in Azure
- Update GCP machine configuration
I think this would work well enough.
Alternatively, we may want to keep people hitting Azure until all the GCP machines are reconfigured, which would look like:
- Promote postgresql in GCP to the primary
- Update GCP machine configuration
- Update the DNS
- Wait 300 seconds
- Turn off postgresql in Azure
We could also reconsider turning off postgresql in azure. Is it at all necessary?