<!-- ISSUE TITLING: use the form "YYYY-MM-DD: briefly describe problem" --> <!-- ISSUE LABELING: Don't forget to add labels for severity (S1 - S4) and service. if the incident relates to sensitive data, or is security related use the label ~security and mark it confidential. --> ## Summary <!-- Leave a brief headline remark so that people know what's going on. It is perfectly acceptable for this to be vague while not much is known. --> During implementation of production#2648, snapshots were initiated on all database servers concurrently, under the mistaken impression that the snapshot process was a transparent background transaction via GCP API, and would not impact the running nodes. ## Timeline <!-- Try to capture in this section, among other events: - Time estimation for when the errors started - typically before the incident was declared. - When the incident was declared. - If other teams had to be engaged, when the right Subject Matter Expert (SME) - able to effectively work on the incident mitigation - was engaged. - When the CMOC sent first comms for this incident. - When the incident was mitigated. - When the incident was fully resolved. - A link to the original PagerDuty incident page, if any. - ... --> All times UTC. `2021-01-16` - `23:36` - @craig initiates snapshots on all patroni nodes - `23:38` - begin seeing [error spikes](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/3338#note_487129599) - `23:39` - multiple alerts received for service disruption - `23:40` - patroni invalidates the lock by `patroni-06` and [begins failover](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/3338#note_488372047) - `23:41` - `patroni-03` [becomes the new primary](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/3338#note_488372047) - `23:42` - [increased error rates fall off](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/3338#note_487129599) - `23:47` - @dawsmith reports [system degradation](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/2648#note_487121073), and cancellation of some snapshot requests - `23:56` - @craig declares incident in Slack. ## Corrective Actions <!-- - _List issues that have been created as corrective actions from this incident._ - _For each issue, include the following:_ - _<Bare Issue link> - Issue labeled as ~"corrective action"._ - _Include an estimated date of completion of the corrective action._ - _Include the named individual who owns the delivery of the corrective action._ - _If an incident review was completed, use Lessons Learned as a guideline for creation of Corrective Actions_ --> 1. https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/12355 - Update Patroni/Postgres runbooks to clarify GCP snapshot best practices. 1. https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/12363 - Add checklist for change management reviewers and approvers to runbooks with reference in handbook 1. https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/12518 - Schedule call with Google to talk more about IO sensitivites. ---- # Incident Review ---- <!-- The purpose of this Incident Review is to serve as a classroom to help us better understand the root causes of an incident. Treating it as a classroom allows us to create the space to let us focus on devising the mechanisms needed to prevent a similar incident from recurring in the future. A root cause can **never be a person** and this Incident Review should be written to refer to the system and the context rather than the specific actors. As placeholders for names, consider the usage of nouns like "technician", "engineer on-call", "developer", etc.. --> ## Summary <!-- _A brief summary of what happened. Try to make it as executive-friendly as possible._ _example: For a period of 19 minutes (between 2020-05-01 12:00 UTC and 2020-05-01 12:19 UTC), GitLab.com experienced a drop in traffic to the database. 507 customers saw 2,342 503 errors over this 19 minute period. The underlying cause has been determined to be a change to the PgBouncer configuration (https://gitlab.com/gitlab-com/gl-infra/production/-/issues/XXXX) which caused the total number of connections to be reduced to 50. This incident was then mitigated by rolling back this PgBouncer configuration change. --> 1. Service(s) affected: ~"Service::Patroni" 1. Team attribution: ~"team::Reliability" 1. Time to detection: 1-2 minutes 1. Minutes downtime or degradation: 4-5 minutes <!-- _For calculating duration of event, use the [Platform Metrics Dashboard](https://dashboards.gitlab.net/d/general-triage/general-platform-triage?orgId=1) to look at appdex and SLO violations._ --> ## Metrics <!-- _Provide any relevant graphs that could help understand the impact of the incident and its dynamics._ --> {{https://gitlab.com/gitlab-com/gl-infra/production/uploads/717d15e470cb6b71217014c686640ea5/Screen_Shot_2021-01-16_at_4.48.01_PM.png}} {{https://gitlab.com/gitlab-com/gl-infra/production/uploads/90fc837f6035d69dcbce71bca9ac07bc/Screen_Shot_2021-01-16_at_4.47.51_PM.png}} ## Customer Impact 1. **Who was impacted by this incident? (i.e. external customers, internal customers)** 1. All active users during the incident 2. **What was the customer experience during the incident? (i.e. preventing them from doing X, incorrect display of Y, ...)** 1. Some would have received 5xx errors for transactions requiring database writes 1. Page refreshes spot checked during the incident responded normally 3. **How many customers were affected?** 1. Unknown, see below for details on requests vs error rates 4. **If a precise customer impact number is unknown, what is the estimated impact (number and ratio of failed requests, amount of traffic drop, ...)?** 1. Full thread discussing impact: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/3338#note_487128596 1. Rails 1. approximately 2000 5xx errors with a spike approximately between 23:38:30 and 23:42:00 1. 1,521,180 requests logged. so a poor calc 2019 / 1,521,180 =~ 0.133% error rate 1. Workhorse 1. 13757 / 481335 =~ 2.86 % error rate 1. Aggregate 1. If we zoom in to just the 1hr from 23:00 to 00:00UTC, .78 availability points were used. ## What were the root causes? ["5 Whys"](https://en.wikipedia.org/wiki/Five_whys) 1. Why did we see errors across the fleet? 1. The primary patroni node initiated a failover 1. Why did patroni initiate a failover? 1. The data disk experienced excessive IO latency 1. The node experienced a loss of network connectivity to consul at the same time, either/both prompting the failover 1. Why did the node experience high IO latency / lose network connectivity 1. A snapshot initiated on the Postgres data volume caused a massive spike in IO latency; presumably due to consuming all available network bandwidth for the network-attached persistent disks 1. Why did we snapshot the volume 1. As a safety measure in preparation for a change to the related filesystem 1. The engineer planning the change was not aware of the underlying architectural aspects of persistent disks (network attached, consuming shared bandwidth from VM instance), and therefore the potential for causing missed heartbeats/health checks ## Incident Response Analysis 1. **How was the incident detected?** 1. Alert flood indicating downstream impacts 1. **How could detection time be improved?** 1. Not really applicable in this case, as the cause and effect were demonstrably linked, and clearly related; we knew immediately that the singular, last action resulted in a severe degradation or interruption of service on the primary patroni node. 1. **How was the root cause diagnosed?** 1. During the incident, demonstrable cause/effect identified the act of snapshotting the disk as the trigger event 1. Subsequent examination of metrics and logs reaffirmed the causal link between snapshot and failover, though the metric data seemed somewhat contradictory 1. Later we realized/remembered that disks are accessed using shared network bandwidth provisioned to an instance, which resulted in increased IO latency and ultimately caused disk access to stall on the primary, triggering the failover. (Need further specifics/external references on the mechanics involved here, if possible) 1. **How could time to diagnosis be improved?** 1. Much like detection, this was not really applicable, as the cause and effect were immediately, demonstrably linked, and clearly related 1. That being said, we did not realize at the time that the impact extended past degradation and prompted an actual failover during the actual incident. We canceled snapshots in two of the three zones when we saw signs of more serious degradation than anticipated, but only noticed the failover later when we proceeded with the subsequent steps in the change plan 1. **How did we reach the point where we knew how to mitigate the impact?** 1. Same as before, straightforward cause & effect 1. **How could time to mitigation be improved?** 1. N/A 1. **What went well?** 1. The additional visibility incurred by executing a production change on such a high-risk portion of the infrastructure meant that we had many additional eyes on hand to keep an eye on metrics and application testing simultaneously. Even with a direct causal link so evident, this dramatically shortened the time to realize that no significant negative impacts had been incurred, and that we really could safely proceed with executing the change plan. 1. This incident highlighted some areas for improvement in our change process ## Post Incident Analysis 1. **Did we have other events in the past with the same root cause?** 1. We have seen similar issues with snapshots resulting in patroni failovers 1. **Do we have existing backlog items that would've prevented or greatly reduced the impact of this incident?** 1. Not at the time, no. We have logged corrective actions to further document the underlying architectural factors and impacts of snapshots on high-IO volumes 1. **Was this incident triggered by a change (deployment of code or change to infrastructure)? If yes, link the issue.** 1. https://gitlab.com/gitlab-com/gl-infra/production/-/issues/2648 ## Lessons Learned <!-- _Be explicit about what lessons we learned and should carry forward. These usually inform what our corrective actions should be._ _example:_ 1. The results of refactoring activities around our integration tests should be reviewed. (i.e we had 619 tests before refactor but 618 after.) 2. Our tooling to dedupe alarms should have integration tests to ensure it works against existing and newly added alarms. --> Initial points to improve: - Don't GCP-snapshot a PG Leader that is up and running. - Tech reviewers have to review every single change/addition to the CR issue description, before it is approved by a manager/director. - The Datastores team should socialise in detail how the Backups and Snapshots work in our Production Database. - Wait for the start of the change (and the calendar invite) to start any actions related to it. ## Guidelines * [Blameless RCA Guideline](https://about.gitlab.com/handbook/customer-success/professional-services-engineering/workflows/internal/root-cause-analysis.html#meeting-purpose) ## Resources 1. If the **Situation Zoom room** was utilised, recording will be automatically uploaded to [Incident room Google Drive folder](https://drive.google.com/drive/folders/1wtGTU10-sybbCv1LiHIj2AFEbxizlcks) (private) ## Incident Review Stakeholders <!-- "Immediately following the incident: The incident review is started in the original incident issue and the EOC and IMOC are assigned. IMOC and EOC invite stakeholders for involvement in authoring the incident review via an @ mention of their GitLab handle in the incident issue." https://about.gitlab.com/handbook/engineering/infrastructure/incident-review/#incident-review-timeline - @ mention any additional stakeholders below. This could include engineers, engineering managers, engineering directors, quality managers and directors, product managers, technical account managers etc. - Use the product category page (https://about.gitlab.com/handbook/product/product-categories/) to find appropriate stakeholders and the org chart (https://about.gitlab.com/company/team/org-chart/) to find line management representation. - Please ensure that director level management are included on S1 incidents, and let them know that representation is mandatory. --> 1. @albertoramos 1. @Finotto 1. @NikolayS 1. @ahachete 1. @emanuel_ongres 1. @craig 1. @dawsmith 1. @brentnewton