Discussion: Streamline instrumentation code by implementing `prometheus.Collector` in Praefect
There are a few problems with how we do metrics in Praefect:
- The metrics being always registered in the global
prometheus.DefaultRegisterer
makes it difficult to test them.- The global metrics can be accessed but their contents are affected by other tests.
- The metrics constructed from configuration can't be accessed by multiple tests as they would error out with a duplicate registration as they are registered every time they are constructed.
- This leads us to use mock metrics in tests to avoid nil pointer panics on uninitialized metrics. Alternatively, we have to nil check the metrics every time. Other alternative is to by default have an unregistered metric but that can be confusing as one would expect the metric to just work. Using mock metrics also means that we are not testing the whole implementation, including the metric names and labels.
- The metrics can be inconsistent during a scrape. In SQL elector, if a scrape happens concurrently with
updateMetrics
when primary has changed, we could end up with an inconsistent view where prometheus sees every node as a secondary or it sees both the new and the old primary as primaries. - We do not have a a time series in Prometheus before the first observation for the label combination is done. As an example, say we have
rpc_count{node="gitaly-1"}
and we graph it in Grafana. On a fresh start, the timeseries won't be present in Prometheus as we never made any call toRPCCount.WithLabelValue("gitaly-1").Inc()
. Only after we've done such a call, will the time series show up. This means that Grafana doesn't show anything for the graph before we do some increments. The plot missing from Grafana can be mistaken for the query being wrong and can be confusing.
To address all of those points, we could implement prometheus.Collector
interface on our types. Please have a look at an example implementation in !2211 (closed) which has some comments on how the points are fixed.
Edited by 🤖 GitLab Bot 🤖