Gracefully shutdown Gitaly nodes
Problem to solve
Servers need to be shutdown from time to time, to apply security updates, upgraded, or resized. There is no process to do this safely to a primary node.
Simply shutting down a node with active read operations like clones could cause deployments and tests to fail, and otherwise be an inconvenience since clones are not presumable.
Further details
Proposal
It should be possible to shutdown Gitaly gracefully which will cause it reject new connections.
Existing read operations should be allowed to complete within some configurable window like 1 hour. As soon as the read operations are complete the shutdown can proceed.
Write operations can be ignored since writes happen as a transaction, and are being written simultaneously to other nodes. In fact, new write operations to the node that is being shutdown can continue until the process is killed because other nodes will still suceed.