Investigate prometheus proxy speed

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

• Close this issue

Problem Statement

Requests to our prometheus proxy are very slow with the 99th percentile often greater than 1 sec. This effect is possibly exaggerated because each dashboard request we make results in multiple queries to the proxy.

See grafana charts https://dashboards.gitlab.net/d/thYzurImk/rails-controllers?orgId=1&var-action=Projects::Environments::PrometheusApiController%23proxy.json&var-database=influxdb-01-inf-gprd&from=now-12h&to=now

Reach

There are currently very few users that use metrics, but as this increases, this could potentially become a large problem.

Effort

It seems most of our time is spent in Prometheus::ProxyService#execute. My guess is we're waiting for query results from the prometheus server. More investigation is needed.