Stage group apdex-ratio recording is erratic
Coming from gitlab-org/gitlab#357846 (comment 932251903)
When looking at a months worth of data, Grafana plots 1 data-point per 45 minutes ($__interval) and it takes the worst case scenario in that case (min_over_time). So comparing the min_over_time graph, with the metrics that source it (success/total) shows this:
The top graph is what Grafana plots, the bottom is the source metrics for that recording. This shows that there might not have been a performance degradation, but rather something causing the ratio calculation to look erratic (bottom graph). When then taking a minimum over 45m, this caused it to look like it does. These ratios are only used for displaying, not for the error budget calculation. This also explains why it wasn't visible in the SLI panels below the stage group aggregation.
I'm still trying to find the cause of this bounciness in the ratio recording at the time it started.