Testhelper for inspecting collected Prometheus metrics
Often times, Prometheus metrics in Gitaly are tested manually via GDK. This makes metric tests harder to reproduce and increases the likelihood that the intended behavior of those metrics will break. To help enforce metric behavior, a test helper for use in unit tests would be beneficial. An ideal test helper would avoid running a Prometheus listener to make the test simpler and lighter weight. It would also be flexible and dynamic to allow a single interface for fetching various types of metrics (e.g. counters and gauges).
An existing PoC for such a test helper exists in !1362 (merged): https://gitlab.com/gitlab-org/gitaly/blob/570b169dda5c9daaa38164c4a0fc769831f338b1/internal/cache/walker_test.go#L66-102
In that PoC, the default Prometheus collector for the process is periodically queried for a specific metric family. Once the metric family is encountered, the specific labeled metrics are found. Finally, the metric values are compared for an expected value. If the expected values do not appear within a specific timeframe, the test fails.
Generalizing the above pattern might look something like this:
package testhelper
type MetricComparison interface {
IsMetric() bool // able to determine if metric is of the desired type (e.g. counter, counter-vec, gauge)
IsValue() bool // able to determine if metric is at a desired value (e.g. equal to number, greater than number)
}
type PromQuery struct {
MetricFamily string
ExpectedMetrics []MetricComparison
}
func (pq PromQuery) PollCounterVecMetric(t testing.TB, pollInterval time.Duration) bool {}
Another useful and simpler pattern may be a simple fetch API for a specific metric type:
package testhelper
// CollectCounterVecMetric returns nil if the defined counter cannot be found
func CollectCounterVecMetric(t testing.TB, metricFamily string, label *io_prometheus_client.LabelPair) *io_prometheus_client.Counter {}