Incident Review: 2024-10-08: Pipelines not completing

Key Information

Metric Value
Customers Affected 25 (8 Oct) and 8 (9 Oct) -- see sheet in internal comment
Requests Affected Estimated 150k sidekiq jobs (up to 135M)
Incident Severity severity2
Start Time 2024-10-07 20:50 UTC
End Time 2024-10-09 10:02 UTC
Total Duration 37h 12m
Link to Incident Issue #18676 (closed)

Summary

During the incident, sidekiq middleware incorrectly routed certain queries to a replica db that should have been sent to the primary db, causing those queries to see slightly stale data (missing very recently committed transactions). This introduced a race condition where very recent commits may not yet have reached the replica being queried -- a stale read bug.

Not all sidekiq jobs are sensitive to this race condition, but a few are.

Sidekiq jobs that update CI pipeline status based on their CI jobs' status were prone to that race condition. The most common user-facing symptoms were:

  • Stuck CI pipeline: Random CI pipelines would get perpetually stuck in the running status after its last CI job finished. Similarly, a multi-stage CI pipeline would randomly not advance to the next stage when the preceding stage's jobs finish. These symptoms occurred if the PipelineProcessWorker sidekiq job queried a stale replica db that had not yet received the latest status update for that pipeline's CI jobs.
  • Stuck merge request: Creating a merge request could randomly fail to start its associated CI pipeline, perpetually displaying Your merge request is almost ready! in the web UI but never starting the pipeline's jobs.

CI minutes accounting was not affected by this bug. CI minutes are consumed only for time spent executing a CI job on a CI runner. CI jobs stuck in created or pending status do not cost CI minutes, and the CI pipeline's status has no bearing on that accounting.

During this incident investigation, we discovered what turned out to be an unrelated pathology consuming excessive redis CPU time. We will have separate corrective actions for that as well, but it is fundamentally a separate concern and was not a contributing cause to this incident.

Details

The incident was triggered on notices that pipelines and merge requests are stuck.

Initially, the problem looked like Sidekiq saturation caused by a long running transaction. The EOC terminated the query and we observed sidekiq apdex recovert. The incident was marked as mitigated.

Due to only 2 reported customers and a recovering system, the incident was downgraded to S3.

Round 2

Customer reports started coming in. The investigation continued by yielded no results

About October 9, 2024 at 16:39:24 UTC, the EOC was alerted of Connection Timeouts in the Redis Cluster. It was caused by the traffic from a single user so their repository has a huge of branches with a high update rate. That pushed the pressure on Redis. The user was blocked which led to the Redis Cluster recovery.

The incident was marked as mitigated subsequently.

Round 3

Despite the pending job queue recovering to levels prior the incident, there are still customer complaints of stuck pipelines. The incident got set to active again.

This time, we discovered that some workers read from the replica (analysis). When it checks for need_processing?, the replica returns stale data. We are digging more to see if it forms a pattern and why replica is used even if the worker uses data_consistency: always.

We opened this MR to reverse the potential MR. While waiting for it to land on gprd, we've enabled the feature flag associated with the MR that will prevent new pipelines and Merge Requests from being impacted. We don't want to retry pipelines and MRs automatically. Pipelines and MR that were previously stuck will need to be recreated by the above workaround.

With the MR now deployed to production, we have marked this incident as IncidentResolved

Outcomes/Corrective Actions

  1. https://gitlab.com/gitlab-org/gitlab/-/issues/498399+
  2. Infradev: Zoekt Worker queue stuck at 1k lenght (gitlab-org/gitlab#498330 - closed)
  3. Increasing redis-cluster-repo-cache apdex threshold to 0.9999: gitlab-com/runbooks!7977 (merged)
  4. Older pipelines with Pipeline status not matchi... (gitlab-org/gitlab#502703 - closed)

Learning Opportunities

What went well?

  1. Upon identifying the root cause, we were able to quickly remedy the symptom by enabling the feature flag while the revert MR is preparing to deploy.
  2. gitlab.com on a whole was able to operate (albeit degraded) with :always workers reading from replicas (until a write happens, then it sticks to the primary for subsequent reads).

What was difficult?

  1. It took us quite some time to narrow down the root cause due to other red herrings during the incident window like a long running transaction that degraded the database, redis-cluster-repo-cache degradation due to an increase in shard cpu usage.
  2. The exactly onset of the symptom was not obvious initially during the incident. The incident was marked as mitigated and reverted to active a few times due to the lack of a clear indicator of the problem. Later on, when we identified that workers with :always data consistency were using replicas in this chart, we were able very quickly to narrow down to a particular deployment and pinpoint the root cause. The real impact of the bug was difficult to identify as it would surface most obviously in pipelines and MRs.

Review Guidelines

This review should be completed by the team which owns the service causing the alert. That team has the most context around what caused the problem and what information will be needed for an effective fix. The EOC or IMOC may create this issue, but unless they are also on the service owning team, they should assign someone from that team as the DRI.

For the person opening the Incident Review

  • Set the title to Incident Review: (Incident issue name)
  • Assign a Service::* label (most likely matching the one on the incident issue)
  • Set a Severity::* label which matches the incident
  • In the Key Information section, make sure to include a link to the incident issue
  • Find and Assign a DRI from the team which owns the service (check their slack channel or assign the team's manager) The DRI for the incident review is the issue assignee.
  • Announce the incident review in the incident channel on Slack.
:mega: @here An incident review issue was created for this incident with <USER> assigned as the DRI.
If you have any review feedback please add it to <ISSUE_LINK>.

For the assigned DRI

  • Fill in the remaining fields in the Key Information section, using the incident issue as a reference. Feel free to ask the EOC or other folks involved if anything is difficult to find.
  • If there are metrics showing Customers Affected or Requests Affected, link those metrics in those fields
  • Create a few short sentences in the Summary section summarizing what happened (TL;DR)
  • Use the description section to write a few paragraphs explaining what happened
  • Link any corrective actions and describe any other actions or outcomes from the incident
  • Consider the implications for self-managed and Dedicated instances. For example, do any bug fixes need to be backported?
  • Add any appropriate labels based on the incident issue and discussions
  • Once discussion wraps up in the comments, summarize any takeaways in the details section
  • Close the review before the due date
Edited by Rehab