Automatically set Prometheus step interval
Problem to solve
Currently, I believe, we retrieve the full resolution of the Prometheus query. So we get a large amount of data back per chart, finer than what we actually need to render a graph.
This can increase the load on the Prometheus server, make our pages quite heavy to load (hundreds of KB), and generally reduce performance. We should see if we can optimize this.
Further details
Proposal
Prometheus supports the concept of a step interval, where the service will attempt to return a desired quantity of data points. Other charting solutions use this, like Grafana, to reduce the data returned especially over longer intervals.
We should support this as well, otherwise zooming out to longer time spans will send far too much data. (For example 1 or 2 weeks)
To achieve this, we should automatically set the step interval to match the resolution of a data point per pixel on screen, if possible. We should also attempt to align on the Prometheus instance's scan interval, to reduce noise and jumpiness. For an example of how both of these things are done, we could take a quick look at Grafana.