You need to sign in or sign up before continuing.
Improve "Bring a demoted primary back online" documentation
Bring a demoted primary back online https://docs.gitlab.com/ee/gitlab-geo/bring-primary-back.html
After the 13 Feb DR demo, I attempted to demote the primary and ran into multiple issues, without success (the repositories were not replicating properly). This might eventually be ok since we are adding repository checking.
In the meantime, some improvements can be made to the doc:
-
Warn about potential pitfalls, e.g. unblocking the primary before the DNS has replicated everywhere might allow users to push to the former primary again. -
The "Setup the database replication step" https://docs.gitlab.com/ee/gitlab-geo/bring-primary-back.html#configure-the-former-primary-to-be-a-secondary links to the DB replication doc, but these instructions don't work in this context. It needs it's own documentation. -
When you add a secondary node in /admin/geo_nodes, the current primary starts talking to itself over the secondary URL, and a job keeps trying to create another geo_nodes record but fails -
The former primary’s gitlab.rb
must be modified first, and reconfigured, before configuring DB replication -
Note that the DB will be wiped before replication can start (and/or open an issue to allow replication to pick up without wiping the former primary DB)
-
-
For now, I think it makes sense to emphasize the "add a secondary from scratch" option a bit more. It would be cleaner, safer, and you may want to keep the former primary around just in case there is data there you need, or to figure out what happened. Perhaps the former primary's data is in an inconsistent state, perhaps the replication wasn't 100% perfect during planned/unplanned failover.
Edited by Michael Kozono