POC: Adding custom healthcheck service
This is a proof of concept of using our own healthcheck endpoint that we can control. In this MR, there's a simple in memory tracker of how many read errors / write errors have happened in a 10 second interval. We then have our healthcheck endpoint return SERVING
or NOT_SERVING
based on this.
The advantage of this, is that we can encapsulate all of our "health" logic into this RPC and use whatever heuristics we'd like for praefect to determine whether or not it wants to do a failover. So then the logic lives in gitaly and keeps the parefect node manager simple.