Update consul server restart to check for quorum
If a consul cluster loses quorum, it can be difficult to recover
If we happen to run reconfigure on all server nodes simultaneously, it's likely they will all try and restart at the same time, breaking quorum, and requiring manual intervention to recover.
Off the top of my head, I have a couple of ideas
-
Put in a pre-check to ensure that restarting won't break quorum. Plus a locking process during restart to ensure that no other nodes try and restart at the time.
-
Add a warning to documentation.