Enable Prometheus collection of InfluxDB metrics

Need

Provide a way to collect internal application metrics for Prometheus scraping.

Approach

GitLab has some support for pushing metrics from Rails to InfluxDB via push UDP packets. Using an InfluxDB metric relay on localhost provides a way to send these metrics to the Prometheus influxdb_exporter.

• #1042 (closed) Install Prometheus influxdb_exporter to listen on localhost.
• #1041 (closed) Install and configure influxdb-relay to tee traffic to prometheus exporter and InfluxDB servers
• #1043 (closed) Transition traffic from InfluxDB to localhost relays

Benefit

• Provides a stable "tee" of metrics traffic to existing InfluxDB database.
• Easily roll out and roll back.
• Improves reliability of transport of metrics to InfluxDB.

Competition

• Switch traffic directly to influxdb_exporter Pros:
  • Simple and easy transition. Cons:
  • InfluxDB dashboards need to be replaced ahead of time, this will be difficult.
  • Rollback would result in gaps in the existing InfluxDB database.