<!-- **Please note:** if the incident relates to sensitive data, or is security related consider labeling this issue with ~security and mark it confidential. *** The purpose of this Incident Review is to serve as a classroom to help us better understand the root causes of an incident. Treating it as a classroom allows us to create the space to let us focus on devising the mechanisms needed to prevent a similar incident from recurring in the future. A root cause can **never be a person** and this Incident Review should be written to refer to the system and the context rather than the specific actors. As placeholders for names, consider the usage of nouns like "technician", "engineer on-call", "developer", etc.. --> Incident: gitlab-com/gl-infra/production#2087 ## Summary <!-- _A brief summary of what happened. Try to make it as executive-friendly as possible._ --> On May 7th, 2020 between 04:26AM UTC - 04:58AM UTC, about.gitlab.com went down and showed: ``` This XML file does not appear to have any style information associated with it. The document tree is shown below. NoSuchKey The specified key does not exist. ``` 1. Service(s) affected : about.gitlab.com 1. Team attribution : TBD 1. Minutes downtime or degradation : 32 mins <!-- _For calculating duration of event, use the [Platform Metrics Dashboard](https://dashboards.gitlab.net/d/general-triage/general-platform-triage?orgId=1) to look at appdex and SLO violations._ --> ## Metrics <!-- _Provide any relevant graphs that could help understand the impact of the incident and its dynamics._ --> N/A ## Customer Impact 1. **Who was impacted by this incident?**<br> All customers who tried to browse to the about.gitlab.com page<br><br> 2. **What was the customer experience during the incident?** <br> They were getting an XML error: "NoSuchKey" <br><br> 3. **How many customers were affected?**<br> - According to https://log.gprd.gitlab.net/goto/1e16992114f9fe6c4af41a80ea770b0b, there were about **19K** requests that came in for about.gitlab.com. - And According to https://log.gprd.gitlab.net/goto/13ff14b8d7f9b0c5743b4346bd571d4c, there were **2,421** unique remote IP addresses during the time frame. ## Incident Response Analysis 1. **How was the event detected?**<br> - User reported in Slack via: https://gitlab.slack.com/archives/C101F3796/p1588825616259400 - 2 minutes later alert came in: https://gitlab.slack.com/archives/C101F3796/p1588825692259800 2. **How could detection time be improved?**<br> - I don't think it needs improvement. I believe we detected the issue in good amount of time. 3. **How did we reach the point where we knew how to mitigate the impact?**<br> - When a team member pointed to a pipeline deployment which was ongoing 4. **How could time to mitigation be improved?**<br> - It was just a matter of time to wait until deployment finished ## Post Incident Analysis 1. **How was the root cause diagnosed?**<br> - Proximal root cause points to a recent deployment: https://gitlab.com/gitlab-com/www-gitlab-com/-/jobs/541883682 where the logs show there were files being removed including the main `index.html`. However, we need to bring in team member(s) who may have gotten involved with the deployment to get data point on the root cause. 2. **How could time to diagnosis be improved?**<br> - EOC wasn't aware of `index.html` missing right away eve though a general knowledge about a website being hosted from a cloud bucket would most likely need one. Improvement to Runbook: https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/uncategorized/about-gitlab-com.md could help improving MTTD - Created https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/10091 for the above corrective action item 3. **Do we have an existing backlog item that would've prevented or greatly reduced the impact of this incident?**<br> N/A 4. **Was this incident triggered by a change (deployment of code or change to infrastructure. _if yes, have you linked the issue which represents the change?_)?**<br> Yes and yes. ## Timeline 2020-05-07<br><br> * 04:26 - Team member reports an issue with [Customers Portal FAQ](https://about.gitlab.com/pricing/licensing-faq/) * 04:28 - PD alert: Pingdom check check:https://gitlab.com/ is down * 04:29 - PD alert: Pingdom check check:https://about.gitlab.com/ is down * 04:30 - Incident declared from Slack * 04:31 - PD alert: Pingdom check check:http://about.gitlab.com/ is down * 04:33 - PD alert: Firing 1 - [www.gitlab.com](http://www.gitlab.com) is down for 2 minutes * 04:33 - PD alert: Pingdom check check:http://gitlab.org/ is down * 04:41 - PD alert: Firing 1 - Chef client failures have reached critical levels * 04:41 - EOC finds out `index.html` is missing in the GCP bucket * 04:52 - Team member helps finding an [ongoing deployment](https://gitlab.com/gitlab-com/www-gitlab-com/-/jobs/541916490) which should hopefully resolve the issue and also a [recent deployment](https://gitlab.com/gitlab-com/www-gitlab-com/-/jobs/541883682) which removed the index.html. * 04:58 - PD alerts got resolved ## 5 Whys - **Why did we get paged for about.gitlab.com being down?** => Because the about.gitlab.com page wasn't responding properly with 200 and it was reporting with XML error "NoSuchKey". - **Why was the about.gitlab.com returning the XML error?** => Because the index.html was gone. - **Why was the index.html was gone?** => It is not confirmed 100% yet, but it looks like it was because there was this deployment https://gitlab.com/gitlab-com/www-gitlab-com/-/jobs/541883682 which removed the index.html among other files. - **Why did the deployment remove files?** => The artifacts from two of the prior `build_master` jobs were not properly downloaded by the `deploy` job. See details here: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/2088#note_337974367 - **Why didn't the `deploy` job properly download the artifacts?** => Appears to be related to this issue: https://gitlab.com/gitlab-org/gitlab/-/issues/212349 ## Lessons Learned 1. A bad deployment could wipe out essential files that make the website working. 2. A domain knowledge about a bucket-sourced website such as about.gitlab.com would have been helpful <!-- _Be explicit about what lessons we learned and should carry forward. These usually inform what our corrective actions should be._ _example:_ 1. The results of refactoring activites around our integration tests should be reviewed. (i.e we had 619 tests before refactor but 618 after.) 2. Our tooling to dedupe alarms should have integration tests to ensure it works against existing and newly added alarms. --> ## Corrective Actions <!-- - _Use Lessons Learned as a guideline for creation of Corrective Actions - _List issues that have been created as corrective actions from this incident._ - _For each issue, include the following:_ - _<Bare Issue link> - Issue labeled as ~"corrective action"._ - _Include an estimated date of completion of the corrective action._ - _Include the named individual who owns the delivery of the corrective action._ --> - Update runbook: https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/10091 - Add logic to CI config to guard against site outages due to artifact upload errors: gitlab-com/www-gitlab-com#7661 ## Guidelines * [Blameless RCA Guideline](https://about.gitlab.com/handbook/customer-success/professional-services-engineering/workflows/internal/root-cause-analysis.html#meeting-purpose)