Investigate prometheus proxy speed
Problem Statement
Requests to our prometheus proxy are very slow with the 99th percentile often greater than 1 sec. This effect is possibly exaggerated because each dashboard request we make results in multiple queries to the proxy.
Reach
There are currently very few users that use metrics, but as this increases, this could potentially become a large problem.
Effort
It seems most of our time is spent in Prometheus::ProxyService#execute
. My guess is we're waiting for query results from the prometheus server. More investigation is needed.