Incident Review: Redis-sidekiq sentinel loss of redundancy results in increased 500 errors

Incident Review

The DRI for the incident review is the issue assignee.

  • If applicable, ensure that the exec summary is completed at the top of the associated incident issue, the timeline tab is updated and relevant graphs are included.
  • If there are any corrective actions or infradev issues, ensure they are added as related issues to the original incident.
  • Fill out relevant sections below or link to the meeting review notes that cover these topics

Customer Impact

  1. Who was impacted by this incident? (i.e. external customers, internal customers)
    1. External customers
  2. What was the customer experience during the incident? (i.e. preventing them from doing X, incorrect display of Y, ...)
    1. Widespread, erratic behavior for the duration of the Redis outage (30 mins) including, but not limited to, inability to log in via SAML. For a small subset of MRs created during the outage window, inability to view diffs, pipeline status, or to merge continued for several hours.
  3. How many customers were affected?
    1. Best estimate is 16,621. However, the actual effects might have been much larger. See #8203 (comment 1231830402).
  4. If a precise customer impact number is unknown, what is the estimated impact (number and ratio of failed requests, amount of traffic drop, ...)?
    1. Disruption was widespread for the duration of the outage (30 mins). All services using Sidekiq jobs were fully down for this period and there were upsteam effects on Web and API requests.
    2. For several hours afterwards, approximately 4,000 MRs were unusable.

What were the root causes?

  • redis-sidekiq-01-db-gprd was down and hasn't been working for a very long time (> 30 days), we have been running with a quorum of 2.
  • redis-sidekiq-03-db-gprd failed due to a GCP host error, the VM was reset and moved to another node, this was the primary/master node.
  • When the current primary was lost, there was no quorum possible, since the services don't start automatically on boot, we were left with a single replica stuck on a quorum conflict.

Incident Response Analysis

  1. How was the incident detected?
    1. Multiple alerts in the #production channel, starting with Redis client errors and cascading to Error SLO violations on multiple services (e.g. Webservice & API LoadBalancer, Puma)
  2. How could detection time be improved?
    1. Alert when the Sentinel service is down
  3. How was the root cause diagnosed?
    1. Investigation by @f_santos
  4. How could time to diagnosis be improved?
    1. By alerting on a down Sentinel node the incident could have been avoided entirely by ensuring enough nodes up to create a quorum should failover be required.
    2. corrective actions:
  5. How did we reach the point where we knew how to mitigate the impact?
    1. Once it was established that there were not enough machines to elect a primary, it became clear that the failed node had to be restarted.
  6. How could time to mitigation be improved?
    1. The long-tail of mitigation, which took several hours, was caused by MRs not being able to self-recover from a Redis Sidekiq outage. infradev issue Enqueue NewMergeRequestWorker for broken MRs af... (gitlab-org/gitlab#387176 - closed) has been opened to make MRs robust to Redis Sidekiq outages.

Post Incident Analysis

  1. Did we have other events in the past with the same root cause?
    1. https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/9431+
  2. Do we have existing backlog items that would've prevented or greatly reduced the impact of this incident?
    1. An alert was added in Adds alert for sentinel loss of redundancy (gitlab-com/runbooks!2040 - merged) but removed in gitlab-com/runbooks@8105f185 for being too noisy to be useful.
  3. Was this incident triggered by a change (deployment of code or change to infrastructure)? If yes, link the issue.
    1. No, it was triggered by a GCP issue. However, the removal of a (noisy) alert prior to this might have increased the chances the down node was caught before it caused an issue with creating a quorum.

What went well?

  • ...

Guidelines

Edited by John Hope