Geo nodes should react to a change in their primary/secondary/none status without a restart
This comes up a lot in the demo and was also mentioned in a call today in the context of GitLab HA. It will also be a boon to Geo DR.
When we add or remove entries to the geo_nodes table, this affects the machines that implement the referenced Geo node. In particular, secondaries have cronjobs enabled that are not present for the primary, and vice-versa. This is done once at start-up, so if you add a new Geo node to the database, you have to restart all the machines implementing the node before it will act correctly.
This is very weird in a HA situation, and leads to hard-to-debug errors and possible inconsistent state.
We should modify the startup process so that we can cope with arbitrary changes to the geo_nodes table without necessitating a restart.
/cc @jramsay