2022-04-13: Git operations failing for some users

Incident DRI

@nnelson

Current Status

The feature flag audit_event_streaming_git_operations https://gitlab.com/gitlab-org/gitlab/-/issues/357211 was the cause of the errors. Once we detected the issue, the flag was disabled which resolved it.

Summary for CMOC notice / Exec summary:

  1. Customer Impact: Users of the git service are likely encountering intermittent errors Approximately 8% of upload-pack requests (most git operations) failed on gitlab.com for the duration of the feature flag.
  2. Service Impact: ServiceGit
  3. Impact Duration: 20:27 - 21:04 utc (38 minutes)
  4. Root cause: Feature flag https://gitlab.com/gitlab-org/gitlab/-/issues/357211

Timeline

Recent Events (available internally only):

All times UTC.

2022-04-13

  • 20:25 - An engineer enables the audit_event_streaming_git_operations feature flag.
  • 20:26 - Frontend request error rates immediately begin to increase.
  • 20:58 - @nnelson receives multiple pages from multiple alerts being fired simultaneously.
  • 20:58 - @nnelson declares severity1 incident in Slack.
  • 21:00 - @nnelson notes in the incident conference bridge that the site is responding normally and the incident severity should be downgraded.
  • 21:01 - @nnelson pulled up the triage dashboard page and noticed that the error rate increase was highly correlated with a feature flag enablement. This is noted in the incident conference bridge call.
  • 21:01 - @nnelson reaches out to the enabling engineer, informs of the elevated error rate, and invites to the incident conference bridge.
  • 21:04 - The feature flag is disabled.
  • 21:06 - @nnelson downgrades the incident severity to severity2 because the relevant errors are not causing complete outage for any users.
  • 21:12 - @nnelson updates issue description with summary information.
  • 21:12 - @skarbek confirms that disabling the feature flag returned the error rate to normal.
  • 21:12 - Incident is marked as IncidentMitigated and the incident conference bridge is disbanded.

Create related issues

Use the following links to create related issues to this incident if additional work needs to be completed after it is resolved:

Takeaways

  • Something we learned from the incident:
  • Things that were surprising or unexpected.
    • Alerts were not triggered until precisely 30 minutes after the feature flag was enabled and error rates began to increase.
    • So many alerts fired simultaneously that the #production slack channel was flooded with alert messages. This led the On-call engineer to over-react (an over-abundance of caution?) and declare severity level 1 incident.
  • Things that went well during incident response.
    • It was immediately apparent that the gitlab.com site was up and responsive, making it clear that this incident was not a severity1 .
    • It was also immediately clear that a feature flag annotation was highly correlated with the incidence of error rate increase, making it extremely easy to mitigate the situation.

Corrective Actions

Corrective actions should be put here as soon as an incident is mitigated, ensure that all corrective actions mentioned in the notes below are included.

Note: In some cases we need to redact information from public view. We only do this in a limited number of documented cases. This might include the summary, timeline or any other bits of information, laid out in out handbook page. Any of this confidential data will be in a linked issue, only visible internally. By default, all information we can share, will be public, in accordance to our transparency value.


Click to expand or collapse the Incident Review section.

Incident Review

The audit_event_streaming_git_operations feature flag was enabled. This caused front-end request error rates to increase. Thirty minutes later, multiple alerts fired simultaneously, causing the EOC to be receive multiple pages all at once.

  • Ensure that the exec summary is completed at the top of the incident issue, the timeline is updated and relevant graphs are included in the summary
  • If there are any corrective action items mentioned in the notes on the incident, ensure they are listed in the "Corrective Action" section
  • Fill out relevant sections below or link to the meeting review notes that cover these topics

Customer Impact

  1. Who was impacted by this incident? (i.e. external customers, internal customers)
    1. external customers, internal customers, anyone using the git service
  2. What was the customer experience during the incident? (i.e. preventing them from doing X, incorrect display of Y, ...)
    1. Occasional 500 errors.
  3. How many customers were affected?
    1. Less than 5% of all users.
  4. If a precise customer impact number is unknown, what is the estimated impact (number and ratio of failed requests, amount of traffic drop, ...)?
    1. ...

What were the root causes?

  • The git request error rates increase incident was caused by the enablement of the audit_event_streaming_git_operations feature flag.
  • It is not yet clear to this engineer (@nnelson) what the root cause of the error rates increase incident was with respect to the enabled feature flag.

Incident Response Analysis

  1. How was the incident detected?
    1. Multiple alerts for multiple services were triggered and the EOC received multiple simultaneous pages.
  2. How could detection time be improved?
    1. The alerting window might be tightened for at least the service closest to the user interface.
  3. How was the root cause diagnosed?
    1. The gitlab.com triage dashboard was pulled up and a feature flag annotation was observed to be highly correlated with the elevated error rate incidence.
  4. How could time to diagnosis be improved?
    1. The diagnosis suspicion took less than 1 minute to develop.
    2. The diagnosis confirmation required an experimental disablement of the feature flag identified by the initial cursory troubleshooting.
    3. The diagnosis confirmation could have been speedier had feature flag enablement been required to be followed by at least 10 minutes of monitoring relevant services and front-end dashboards for such incidences of error rate increases.
    4. Such a monitoring requirement would have led the enabling engineer to be more immediately available for requested actions such as the attempted experimental mitigation response.
  5. How did we reach the point where we knew how to mitigate the impact?
    1. Diagnostic suspicion followed by confirmation through experimental action.
  6. How could time to mitigation be improved?
    1. Require engineers to remain available to respond quickly to pings for a certain period of time following feature flag enablement.
  7. What went well?
    1. Error rate increase incidence was correlated quickly and correctly to feature flag enablement.

Post Incident Analysis

The feature flag in question was a feature community contribution from JiHu to add more streaming audit events to git operations. It's unclear given how the 500 errors occurred since streaming audit events is a fairly new feature.

The merge request went through groupcompliance to be reviewed, though no performance issues were anticipated at the time.

@dennis will be working with engineers to investigate the root cause of the errors further.

  1. Did we have other events in the past with the same root cause?
    1. We've had a similar situation with performance degradation where logging audit events for git push operation, ultimately resulting in us deprecating and removing the feature planned for 15.0.
  2. Do we have existing backlog items that would've prevented or greatly reduced the impact of this incident?
    1. @dennis: Not currently, this was using a recent feature, streaming audit events, introduced in 14.5 but enabled by default in 14.7, and this is the first type of streaming audit event pertaining to git operations.
  3. Was this incident triggered by a change (deployment of code or change to infrastructure)? If yes, link the issue.
    1. [Streaming Audit Event] Git fetch actions (gitlab-org/gitlab#343984 - closed)
    2. Feature flag issue: https://gitlab.com/gitlab-org/gitlab/-/issues/357211+

What went well?

    1. We quickly discovered a recently enabled feature flag through the Gitlab Triage dashboard page which led to speedy mitigation of the impact, as well as pulling in the engineer involved to further diagnose.

Guidelines

Resources

  1. If the Situation Zoom room was utilised, recording will be automatically uploaded to Incident room Google Drive folder (private)
Edited by Nels Nelson