DRI @reprazent ### Background Several users of the error budgets for stage groups have reached out to us with questions around unexplainable dips in the error budget for their group. During investigations, we've seen that a lot of these dips don't correlate to actual problems we saw in the application, and also could see these problems in the source metrics. We've noticed that some recordings (from Thanos & Prometheus recording rules) don't match their source metrics. In this project, we'll be working on the reliability & observability of the monitoring infrastructure we use for error budgets for stage groups. Scalability is working together with folks from ~"team::Observability" on this, this is just a collection of the work Scalability is focusing on that should improve the situation specifically for error budgets for stage groups. After this work is done, we should have correct error budgets for stage groups all of the time. If something goes wrong on the observability side of things, we'd ideally be alerted before our users notice, which means improving SLIs & capacity planning for these parts of our infrastructure. Related work ongoing in https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/630, specifically https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/14644 ### Status 2023-05-02 The problems with these metrics were caused by increased cardinality because of the introduction of a new service that would serve Rails traffic. So the first step we did in this project was reduce the overall cardinality so the metrics for error budgets for stage groups would correctly aggregate again. Then we worked on improving part of the implementation of how these metrics are recorded by separating them by environment. This allowed us to change the evaluation strategy from `abort` to `warn`. Meaning rule evaluation would fail, and we'd be notified, instead of silently being incorrect. While we built the method to do this, we also applied this to other autogenerated recording rules in Thanos. We also improved SLIs for rule evaluation to be notified when this happens again, instead of being notified by our users noticing incorrect data. We've decided to cancel the addition of a new saturation point that would try to predict too many high cardinality metrics in a recording rule. We did this because we noticed that our new SLI (https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/2204) would alert us of problems, and because there's more work planned with regards to capacity planning in kubernetes (https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/2271#note_1369996867) that would likely improve things for Thanos & Prometheus running there as well. This project is done, error budgets in the last 2 reports have been reliable, and we put SLIs in place to alert us when recordings start to fail again.