Incident Review: Mimir unavailable

INC-5287: Mimir

Generated by Alex Hanselka on 28 Oct 2025 22:29. All timestamps are local to Etc/UTC

Key Information

Metric Value
Customers Affected Internal only
Requests Affected Internal only
Incident Severity Severity 2 (High)
Impact Start Time Tue, 28 Oct 2025 14:52:00 UTC
Impact End Time Tue, 28 Oct 2025 15:27:00 UTC
Total Duration 25 minutes
Link to Incident Issue #20779 (closed)

Summary

Problem: A failure in Mimir ingesters caused loss of quorum, resulting in full service outage and missing metrics.

Impact: From 14:52 to 15:27 UTC, Mimir became unavailable, causing all queries and alerting to fail. This resulted in loss of monitoring visibility, incomplete metrics, and unreliable Grafana dashboards during the outage window. 18 out of 210 ingesters were affected, which led to write failures and increased latency.

Causes: OOM kill on multiple ingesters due to high numbers of in-memory series. See gitlab-com/gl-infra/observability/team#4393 (closed) for more details.

Response strategy: The system recovered on its own. All alerts have since resolved, and services have returned to normal.

What went well?

  1. It was relatively straightforward for me to discover that there was an issue with the ingesters, as we've hit this problem before.
  2. The issue self resolved, and we did find useful options to help mitigate this problem in the future.

What was difficult?

  1. One of the useful mitigation options blocks-storage.tsdb.early-head-compaction-min-in-memory-series uncovered another issue with some of exporters spending some rather old timestamps. This made the option un-usable until we further audit those as it prevented any batch with an offending series being written to Mimir.
  2. We don't have great debugging documentation on Mimir. https://gitlab.com/gitlab-com/runbooks/-/tree/master/docs/mimir doesn't include any details on how to figure out what's going on with the write path or how to fix it, which means that a lot of times, the problem ends up being only fixable by Observability.

Investigation Details

Timeline

Incident Timeline

2025-10-28

14:52:00 Impact started at

Custom timestamp "Impact started at" occurred

15:25:30 Incident reported by Calliope Gardner

Calliope Gardner reported the incident

Severity: Severity 3 (Medium)

Status: Investigating

15:28:25 Severity upgraded from Severity 3 (Medium) → Severity 2 (High)

Calliope Gardner shared an update

Severity: Severity 3 (Medium) → Severity 2 (High)

Mimir is down, which means our alerting and visibility is also unavaliable

15:37:58 Message from Stephanie Jackson

Stephanie Jackson's message was pinned by Alex Hanselka

Mostly didn't restart...

stejacks@atlas~ $ kubectl -n mimir get pods | grep 'ago' | grep ingester
mimir-ingester-us-east1-b-0                     1/1     Running                  1 (25m ago)      36d
mimir-ingester-us-east1-b-34                    1/1     Running                  1 (25m ago)      36d
mimir-ingester-us-east1-b-37                    1/1     Running                  1 (26m ago)      36d
mimir-ingester-us-east1-b-58                    1/1     Running                  1 (25m ago)      36d
mimir-ingester-us-east1-c-11                    1/1     Running                  1 (44m ago)      38d
mimir-ingester-us-east1-c-2                     1/1     Running                  1 (44m ago)      38d
mimir-ingester-us-east1-c-35                    1/1     Running                  1 (42m ago)      38d
mimir-ingester-us-east1-c-38                    1/1     Running                  1 (44m ago)      38d
mimir-ingester-us-east1-c-42                    1/1     Running                  1 (44m ago)      38d
mimir-ingester-us-east1-c-52                    1/1     Running                  1 (25m ago)      38d
mimir-ingester-us-east1-d-19                    1/1     Running                  1 (25m ago)      38d
mimir-ingester-us-east1-d-3                     1/1     Running                  1 (43m ago)      39d
mimir-ingester-us-east1-d-34                    1/1     Running                  1 (43m ago)      39d
mimir-ingester-us-east1-d-41                    1/1     Running                  1 (42m ago)      38d
mimir-ingester-us-east1-d-47                    1/1     Running                  1 (25m ago)      38d
mimir-ingester-us-east1-d-58                    1/1     Running                  1 (24m ago)      38d
mimir-ingester-us-east1-d-60                    1/1     Running                  1 (42m ago)      38d
mimir-ingester-us-east1-d-65                    1/1     Running                  1 (26m ago)      38d

15:39:52 Message from Stephanie Jackson

Stephanie Jackson's message was pinned by Alex Hanselka

Write issue from 14:52 to 15:27. Errors are 'lack of quorum' in the logs. 18 ingesters restarted. Lots of latency and errors, but no clear reason yet.

15:42:54 Image posted by Stephanie Jackson

Stephanie Jackson posted an image to the channel

Problem definitely corresponds to ingesters having an issue: https://dashboards.gitlab.net/goto/af2fcwzfifq4gf?orgId=1

a slack image

15:50:00 Identified at

Custom timestamp "Identified at" occurred

15:50:52 Incident resolved and entered the post-incident flow

Alex Hanselka shared an update

Status: Investigating → Documenting

Between 14:52 and 15:27, Mimir experienced a write outage due to lack of quorum, caused by 18 ingesters restarting. This led to increased latency and errors, and resulted in incomplete or unreliable data powering all dashboards and alerting during this window. Underlying metrics were not created correctly as rule evaluations failed, but the dashboards themselves were not directly broken.

The incident timing aligns with a recent rollout that completed about 40 minutes before recovery, though no direct link has been established. The system has since recovered, and all related alerts have been resolved. There are no indications of further issues in Mimir reads or writes outside the affected window.

A follow-up has been created to investigate the root cause in more detail.

Investigation Notes

Any details you may want to add about the investigation can go here.

Follow-ups

Follow-up

Owner

  1. Figure out why mimir broke: Observability 2 We need to revisit autoscaling with the ingesters, which would allow us to scale to traffic demands more efficiently. Due to their stateful nature and the fact they keep recent data for queries this does have it's own challenges though, and the effort is likely better spent on the next point.: Observability
  2. Migrate to the newer Mimir Ingest Storage architecture. This provides better write scaling and ability to handle sudden traffic bursts or in our case cardinality explosions due many pods rotating from more frequent deployments: Observability.
  3. Audit metrics emitting old timestamps and re-enable blocks-storage.tsdb.early-head-compaction-min-in-memory-series

Review Guidelines

This review should be completed by the team which owns the service causing the alert. That team has the most context around what caused the problem and what information will be needed for an effective fix. The EOC or IMOC may create this issue, but unless they are also on the service owning team, they should assign someone from that team as the DRI.

For the person opening the Incident Review

  • Set the title to Incident Review: (Incident issue name)
  • Assign a Service::* label (most likely matching the one on the incident issue)
  • Set a Severity::* label which matches the incident
  • In the Key Information section, make sure to include a link to the incident issue
  • Find and Assign a DRI from the team which owns the service (check their slack channel or assign the team's manager) The DRI for the incident review is the issue assignee.

For the assigned DRI

  • Fill in the remaining fields in the Key Information section, using the incident issue as a reference. Feel free to ask the EOC or other folks involved if anything is difficult to find.
  • If there are metrics showing Customers Affected or Requests Affected, link those metrics in those fields
  • Create a few short sentences in the Summary section summarizing what happened (TL;DR)
  • Link any corrective actions and describe any other actions or outcomes from the incident
  • Consider the implications for self-managed and Dedicated instances. For example, do any bug fixes need to be backported?
  • Once discussion wraps up in the comments, summarize any takeaways in the details section
  • If the incident timeline does not contain any sensitive information and this review can be made public, turn off the issue's confidential mode and link this review to the incident issue.
  • Close the review before the due date
  • Go back to the incident channel or page and close out the remaining post-incident tasks
Edited by Stephanie Jackson