2026-05-25: Mimir issues, grafana looks very sad.

Mimir issues, grafana looks very sad. (Severity 3)

Problem: Mimir instability made Grafana dashboards largely unavailable for internal meta-monitoring.

Impact: Only internal meta-monitoring dashboards were affected, making Grafana unavailable for internal monitoring. There was no impact to customers or external platform visibility.

Causes: The incident was caused by overly expensive meta-monitoring recording rules that created high-cardinality queries, saturating the system and resulting in backlogs and instability. Metrics remote-write for Cells was initially suspected but was later determined to be unrelated to the root cause.

Response strategy: We increased the max_queriers_per_tenant limit from 10 to 20 for meta-monitoring, which stabilized the system. As an additional safeguard, we disabled metrics remote-write for Cells, though this was unrelated to the root cause. Internal dashboards are now recovering and monitoring is ongoing to confirm stability.

This ticket was created to track INC-10466, by incident.io 🔥