## Incident Review <!-- The purpose of this Incident Review is to serve as a classroom to help us better understand the root causes of an incident. Treating it as a classroom allows us to create the space to let us focus on devising the mechanisms needed to prevent a similar incident from recurring in the future. A root cause can **never be a person** and this Incident Review should be written to refer to the system and the context rather than the specific actors. As placeholders for names, consider the usage of nouns like "technician", "engineer on-call", "developer", etc.. _For calculating duration of event, use the [Platform Metrics Dashboard](https://dashboards.gitlab.net/d/general-triage/general-platform-triage?orgId=1) to look at appdex and SLO violations._ --> **The DRI for the incident review is the issue assignee.** - [ ] If applicable, ensure that the exec summary is completed at the top of the associated incident issue, the timeline tab is updated and relevant graphs are included. - [ ] If there are any corrective actions or infradev issues, ensure they are added as related issues to the original incident. - [ ] Fill out relevant sections below or link to the meeting review notes that cover these topics ### Customer Impact 1. **Who was impacted by this incident? (i.e. external customers, internal customers)** 1. Users on ~"gitlab.com" 2. **What was the customer experience during the incident? (i.e. preventing them from doing X, incorrect display of Y, ...)** 1. Users reported slow requests and not being able to [load their CI/CD pipelines](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8619#note_1333373640), overall degraded performance on Gitlab.com 3. **How many customers were affected?** 1. Everyone on ~"gitlab.com" 4. **If a precise customer impact number is unknown, what is the estimated impact (number and ratio of failed requests, amount of traffic drop, ...)?** 1. ... 5. **Duration** 1. It took us about **~1hour** to figure out where the problem as coming from. 2. `~3hours` - `14:15 utc - 17:57 utc` ### What were the root causes? We introduced a [bug fix](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/114426) which was **unlocking a lot of pipelines** cascading to unlocks more job artifacts afterwards. This bug fix had a slow query that updates hundreds of rows on `ci_pipelines` and `ci_job_artifacts` table and increased dead tuples on these tables. This caused a datatase saturation leading to a large increase in 500s site-wide and degradation of ~"gitlab.com". ### Incident Response Analysis 1. **How was the incident detected?** 1. Alert on replication lag in CI database https://gitlab.slack.com/archives/C101F3796/p1680101410420179 2. Incident was declared https://gitlab.slack.com/archives/C101F3796/p1680102335670589 2. **How could detection time be improved?** 1. ... 3. **How was the root cause diagnosed?** 1. [Increased pg_stat_activity from the Sidekiq worker was detected](https://gitlab.slack.com/archives/C050S6W2NJ1/p1680104297497609) This worker was then traced back to the MR that introduced it. 4. **How could time to diagnosis be improved?** 1. 5. **How did we reach the point where we knew how to mitigate the impact?** 1. Once the MR that introduced the worker was introduced, rollback was started. 6. **How could time to mitigation be improved?** 1. Feature flagging the new worker so that we could revert to the previous worker without having to rollback. ### Post Incident Analysis 1. **Did we have other events in the past with the same root cause?** 1. None that we are aware of. 1. **Do we have existing backlog items that would've prevented or greatly reduced the impact of this incident?** 1. No as we accumulated a lot of artifacts over the years and https://gitlab.com/gitlab-org/gitlab/-/merge_requests/114426+ was supposed to fix this going forward. We did not anticipate well how big the number of accumulated artifacts would be which is really hard to determine. 1. **Was this incident triggered by a change (deployment of code or change to infrastructure)? If yes, link the issue.** 1. [bug fix](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/114426) to fix these issues https://gitlab.com/gitlab-org/gitlab/-/issues/387087, https://gitlab.com/gitlab-org/gitlab/-/issues/266958 ### What went well? <!-- Use this section to highlight what went well during the incident. Capturing this helps understand informal processes and expertise, and enables undocumented knowledge to be shared. _example:_ 1. We quickly discovered a recently changed feature flag through the event log which enabled fast mitigation of the impact, as well as pulling in the engineer involved to further diagnose. 2. We escalated through dev escalations, which brought in Person X. They knew that Person Y had expertise with the component in question, which enabled faster diagnosis. 3. Person Z discovered a misconfiguration in component A by using a methodology they had used previously. This method was not known by anyone else involved in the incident, and that contribution was crucial to understanding the underlying mechanism of the incident. --> - SRE did a great job at [identifying the worker](https://gitlab.slack.com/archives/C050S6W2NJ1/p1680104297497609) causing the degradation - We [communicated](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8619#note_1333555120) to users once the [fix](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/116144) was deployed and things went back to normal - Database replica on new N2 hardware [handled read-only queries on its own](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8621#note_1335076760) very well for the period that it was up. - `main`/`ci` database separation limited the impact to `CI` queries. ### Guidelines * [Blameless RCA Guideline](https://about.gitlab.com/handbook/customer-success/professional-services-engineering/workflows/internal/root-cause-analysis.html#meeting-purpose)