Technical debt: Improve release information dashboard latency
Context
Following up from #20174 (closed).
The "delivery: release information" dashboard currently has a delay between when the metric is created/emitted, and when the information displays. This is most likely due to querying many time series over a relatively large time range (22 days currently).
This is set up this way because how the panels currently calculate for the release information is by fetching the latest emitted metric throughout the monthly and patch release process. The "Open" status metric for a monthly release is created when the last monthly release is published, then there's a maximum of 22d gap to when the next "Announced" status metric for that release gets created.
Metric creation date | Version (label) | Release Date (label) | Status (value) | Description |
---|---|---|---|---|
2024-03-21 | 16.11 | 2024-04-18 | Open | Created after 16.10 is published |
2024-04-12 | 16.11 | 2024-04-18 | Announced | During initial preparation day for 16.11 |
Notice the time gap between the two creation dates of the metrics. This is why we have the time range of 22 days. Up to the point on 04-12 when the status gets changed from "open" to "announced", the status panel should be able to fetch metric from 03-21 (which only lingers for a few days) and be able to display "open". This gap is a bit less with the patch release, since we release that two times a month.
However, by fetching metrics for the past 22 days and calculating the values from them takes a lot of computation on the grafana side, which slows down the dashboard (known issue on grafana).
This issue is to implement a solution to calculate from a shorter time range.
Unfortunately, we can't just "default" to "open" status, because we also need to display the release versions and release dates, which are included in the metric's labels.
Potential solutions
- More frequent emission of metrics
- A less compute-intensive way of fetching metrics from a longer time period
- Migrate dashboard away from prometheus/grafana