[meta] Add metrics instrumentation to GitLab Internals
In order to measure performance improvements for https://gitlab.com/gitlab-org/gitlab-ce/issues/26789 we need to add Prometheus metrics endpoints and instrument the code to provide stats.
There are a number of existing metrics sources and there are some gaps. We first need to identifiy the gaps.
- Identify gaps where we have no metrics to show performance.
- Add Prometheus client library support where possible.
- Replace existing metrics (ie InfluxDB) with Prometheus native methods.
- Add exporters for 3rd party software without native Prometheus support.
- Add log-based metrics generators where necessary (mtail).
- Add blackbox probing for end-to-end testing.
Adding instrumentation with Prometheus provides us the ability to measure our performance gains, and alert on performance degradation.
Some other approaches are possible, and may be useful as temporary stop-gaps until native support is completed.