Utilize consul for Prometheus service discovery in HA deployments
Problem statement
Today with single-node Omnibus deployments, self-monitoring is enabled and easy to use. We don't however have a pattern for deploying and configuring self-monitoring in an HA environment, with a similar level of ease. Users have to:
- Set up a standalone Prometheus/Grafana node
- Manually configure the Prometheus scrape list to scrape each exporter on each HA node. If an IP changes, this config needs to be modified. In some cloud providers you could leverage a service discovery method, like EC2 SD, but this is not a general purpose solution.
Proposed solution
We should consider writing the various nodes/exporters into Consul so it can be discovered by one (or multiple) Prometheus instances automatically.
This would provide an easier method for our customers to set up monitoring for their HA instances.
To achieve this, we should have a flag which would trigger:
- A process which registers each node/exporter with Consul on startup/shutdown (Consul agent?)
- Sets the exporters to be available beyond localhost
- Enable Consul SD within Prometheus