Praefect failover: what are the failure modes for triggering a failover?

Praefect needs to decide when it will consider an internal gitaly node "down". We need to decide on the criteria.

here are some examples:

1. internal gitaly node cannot be reached x times in a row.
2. internal gitaly node times out x times in a row