Query only available Prometheus metrics
Summary
When visiting Operations > Metrics of a Kubernetes-deployed app I see an increased amount of metric requests to Prometheus after enabling Prometheus API endpoint. Only a fraction of these requested metrics are shown on the Metrics page, though.
Steps to reproduce
- Boot GDK
- Install Kubernetes
- Install Prometheus and deploy AutoDevops app
- Enable feature flag
environment_metrics_use_prometheus_endpoint
via Rails console
Feature.enable(:environment_metrics_use_prometheus_endpoint)
- Open
Network
tab of the JavaScript console and visit Operations > Metrics - Watch many XHR requests for metrics which aren't available/shown
What is the current bug behavior?
The metrics dashboard endpoint (e.g. http://localhost:3001/root/autodevops-deploy/environments/35/metrics_dashboard.json
) returns all metrics.
What is the expected correct behavior?
The metrics dashboard endpoint should only return metrics which are available to reduce the amount of requests to Prometheus.
Possible fixes
The metrics dashboard endpoint should only return available metrics by querying Prometheus' label values utilizing our Prometheus client. This result should be cached via reactive caching.
Here's an example where we match all metrics against currently available metrics: https://gitlab.com/gitlab-org/gitlab-ce/blob/1a3fda63a5f9756cde19bc7e221651b0c33cb5dc/lib/gitlab/prometheus/queries/query_additional_metrics.rb#L74
Refs https://gitlab.com/gitlab-org/gitlab-ce/issues/63789#note_194751893