Implement `/liveness` and `/readiness` probes on Workhorse

Proposal

Currently Workhorse does passthrough of /-/health to the application, and on its own does not implement neither /liveness nor /readiness probes.

We should clearly implement the probes per their definition.

Since the Workhorse is deeply interconnected with Unicorn/Puma, it is used and runs on the same node, or as part of sibling container we cannot consider Workhorse to be ready until the underlying service is responding.

We implemented on Unicorn/Puma the health-checks on separate endpoint. The proposal is to introduce generic readiness_probes_url list of URLs that will be asked for the status.

We will configure readiness_probes_url to point to web_exporter endpoint as part of Omnibus to not make Workhorse to be aware how it runs and what it configures, as in the future the architecture of interconnectivity can change.

Solution

Extend config.toml with readiness_probes_url,
Add /liveness and /readiness on prometheus listen addr as it does not seem to make sense to introduce a new endpoint for that.

References

Edited Oct 23, 2019 by Kamil Trzciński