## Incident Review <!-- The purpose of this Incident Review is to serve as a classroom to help us better understand the root causes of an incident. Treating it as a classroom allows us to create the space to let us focus on devising the mechanisms needed to prevent a similar incident from recurring in the future. A root cause can **never be a person** and this Incident Review should be written to refer to the system and the context rather than the specific actors. As placeholders for names, consider the usage of nouns like "technician", "engineer on-call", "developer", etc.. _For calculating duration of event, use the [Platform Metrics Dashboard](https://dashboards.gitlab.net/d/general-triage/general-platform-triage?orgId=1) to look at appdex and SLO violations._ --> Incident: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8097 **The DRI for the incident review is the issue assignee.** - [x] If applicable, ensure that the exec summary is completed at the top of the associated incident issue, the timeline tab is updated and relevant graphs are included. - [x] If there are any corrective actions or infradev issues, ensure they are added as related issues to the original incident. - [x] Fill out relevant sections below or link to the meeting review notes that cover these topics ### Customer Impact 1. **Who was impacted by this incident? (i.e. external customers, internal customers)** 1. All traffic to GitLab.com. 2. **What was the customer experience during the incident? (i.e. preventing them from doing X, incorrect display of Y, ...)** 1. Users would have received 5XX errors during the outage. 3. **How many customers were affected?** 1. ~100% of traffic 4. **If a precise customer impact number is unknown, what is the estimated impact (number and ratio of failed requests, amount of traffic drop, ...)?** 1. Our CDN reports ~13M requests that resulted in `503` during the outage. ### What were the root causes? An engineer triggered a routine task using our tools on a local development environment that due to an unfortunate series of events and unexpected tool behavior, it resulted in performing a change to production that took down a core component (Consul) of our Postgres HA solution (Patroni). ### Incident Response Analysis 1. **How was the incident detected?** 1. We were alerted immediately by our monitoring & alerting infrastructure. 1. **How could detection time be improved?** 1. We were almost immediately aware of a major incident occurring as soon as the Consul cluster went down impacting on our Patroni clusters. 1. **How was the root cause diagnosed?** 1. As one of the alerts indicated a Patroni failover, it was the first component to be investigated. Patroni wasn’t responding, so we looked at Consul and noticed the Consul servers were missing. We quickly realized that the Consul release in our Kubernetes cluster was missing and proceeded to reinstall it. 1. **How could time to diagnosis be improved?** 1. In this incident, our investigation quickly led us to the Patroni clusters and they were unhealthy. Given their dependence on Consul, we followed the trail and found that the Consul server cluster was missing. 1. **How did we reach the point where we knew how to mitigate the impact?** 1. Once it was clear that the entire consul release had been deleted (all Consul related pods, services, etc were missing), we moved to re-applying the release to production and the Consul cluster quickly came up, and the database clusters recovered. 1. **How could time to mitigation be improved?** 1. The tooling we’re using did not make it straightforward for us to redeploy the Consul release and it required manual intervention. It’s likely that improving our Kubernetes deployment story can help here (see PoC: https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/16875) ### Post Incident Analysis 1. **Did we have other events in the past with the same root cause?** 1. ... 1. **Do we have existing backlog items that would've prevented or greatly reduced the impact of this incident?** 1. We are looking to improve our Kubernetes deployment story, which would greatly reduce the risk of running potentially harmful deployment operations against a production environment from our local environments - https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/16875 1. We are also working on extending our use of Teleport by integrating it with our Kubernetes clusters, and this could be configured to assign a read-only role by default to SREs reducing the risk of accidentally making changes to a production environment - https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/16221 1. **Was this incident triggered by a change (deployment of code or change to infrastructure)? If yes, link the issue.** 1. No. This incident was triggered by an engineer performing routine local development work using a well-known open source tool that behaved in unexpected ways when the context changed from local development to production during a local deploy/rollback operation. We are urgently working on corrective actions to avoid recurrence in future. ### What went well? <!-- Use this section to highlight what went well during the incident. Capturing this helps understand informal processes and expertise, and enables undocumented knowledge to be shared. _example:_ 1. We quickly discovered a recently changed feature flag through the event log which enabled fast mitigation of the impact, as well as pulling in the engineer involved to further diagnose. 2. We escalated through dev escalations, which brought in Person X. They knew that Person Y had expertise with the component in question, which enabled faster diagnosis. 3. Person Z discovered a misconfiguration in component A by using a methodology they had used previously. This method was not known by anyone else involved in the incident, and that contribution was crucial to understanding the underlying mechanism of the incident. --> - Bringing the Consul cluster back up in itself did not take very long. It involved some manual work, but it was carried out relatively quickly. - We were alerted immediately that something major had happened and we identified the root cause within minutes. ### Guidelines * [Blameless RCA Guideline](https://about.gitlab.com/handbook/customer-success/professional-services-engineering/workflows/internal/root-cause-analysis.html#meeting-purpose)