<!-- CONFIDENTIAL ISSUES: Try to keep the primary incident public. If necessary use a vague title. Any confidential information (summary, timeline, findings) should be contained in a linked, confidential issue. --> ## Current Status <!-- Leave a brief headline remark so that people know what's going on. It is perfectly acceptable for this to be vague while not much is known. --> GitLab.com experienced a severe degradation due to an outage on one of our CDN providers fastly.com (see: https://status.fastly.com/incidents/vpk0ssybt3bj for incident details from their perspective). This was a partial degradation for GitLab.com because Fastly is only used for `/assets`, for users who did not have assets cached on their browser the GitLab.com UI will be unusable. This was a full outage for about.GitLab.com. API / Git / Registry / Pages were functioning during the incident. The incident lasted for 59 minutes from 09:58 UTC - 10:57 UTC on 2021-06-08. Summary for CMOC notice / Exec summary: 1. Customer Impact: GitLab.com was degraded (JS and images would often not load. About.GitLab.com experienced a complete outage. 1. Customer Impact Duration: 09:58 UTC - 10:57 UTC (59 minutes) 1. Current state: See `Incident::<state>` label 1. Root cause: ~"RootCause::External-Dependency" (https://status.fastly.com/incidents/vpk0ssybt3bj) ## Timeline <!-- Try to capture in this section, among other events: - Time estimation for when the errors started - typically before the incident was declared. - When the incident was declared. - If other teams had to be engaged, when the right Subject Matter Expert (SME) - able to effectively work on the incident mitigation - was engaged. - When the CMOC sent first comms for this incident. - When the incident was mitigated. - When the incident was fully resolved. - A link to the original PagerDuty incident page, if any. - ... --> **[View recent production deployment and configuration events](https://nonprod-log.gitlab.net/goto/2f2872632ccd39c3895e11290c77c346)** (internal only) All times UTC. `2021-06-08` - `09:58 UTC` - Fastly reports initial incident - [Fastly Incident](https://status.fastly.com/incidents/vpk0ssybt3bj) - `10:15 UTC` - Posted a public status update https://twitter.com/gitlabstatus/status/1402207555724537859 - `10:18 UTC` - A MR to replace the use of Fastly for asset CDN is prepared in https://gitlab.com/gitlab-com/gl-infra/chef-repo/-/merge_requests/85. However, due to a dependency of chef-repo pipeline on an alpine image that is currently also dependent on fastly, we are unable to apply the change - `10:37 UTC` - GitLab.com is intermittently available from various locations. About.gitlab.com is completely unavailable due to Fastly being the first point of entry for the website. - `10:42 UTC` - We are considering manually applying the configuration change outside of our regular pipelines to replace assets dependency on Fastly. The change is being tested on staging.gitlab.com first. - `10:43 UTC` - Chef is updated with CDN_HOST set to https://gitlab.com for the staging environment, running Chef now on the web fleet for testing. After it is applied we should confirm that assets are loading from gitlab.com. - `10:44 UTC` - Fastly reports a fix has been applied - [Fastly Incident](https://status.fastly.com/incidents/vpk0ssybt3bj) - `10:51 UTC` - This google doc has been opened to the world, the infrastructure department has edit access. - `10:52 UTC` - Fastly bypass confirmed on https://staging.gitlab.com/ - `10:54 UTC` - Reports are coming in that fastly is recovering and that about.gitlab.com is fully accessible. GitLab.com assets are loading. - `10:57 UTC` - Fastly reports a fix has been fully rolled out - [Fastly Incident](https://status.fastly.com/incidents/vpk0ssybt3bj) - `10:57 UTC` - I have reverted the change on https://staging.gitlab.com to bypass fastly - `11:01 UTC` - We will be executing the change on Canary to bypass fastly as a test. Canary is taking 5% of traffic, and we have a way of quickly reacting to change. - `11:04 UTC` - MRs for disabling fastly on staging and canary: - Staging https://gitlab.com/gitlab-com/gl-infra/chef-repo/-/merge_requests/86 - Canary https://gitlab.com/gitlab-com/gl-infra/chef-repo/-/merge_requests/87 - `11:15 UTC` - Running $ knife ssh -C1 'roles:gprd-base-fe-web-cny' 'sudo chef-client' to force chef runs on canary to test fastly bypass. - `11:25 UTC` - Canary fleet is bypassing CDN, the web fleet looks stable. We confirmed that this is a valid workaround. We will be running this bypass for the next 30 minutes. - `11:27 UTC` - We are resolving the incident as we are seeing reports of the full recovery. ## Corrective Actions <!-- - List issues that have been created as corrective actions from this incident. - For each issue, include the following: - A one sentence summary of the corrective action. - <Bare Issue link> - Issue labeled as ~"corrective action". - If an incident review was completed, use Lessons Learned as a guideline for creation of Corrective Actions --> - Bake immutable images for critical components. We found an image in our configuration repository that was attempting to install a pip package. - Ensure that we have runbooks to manually apply changes in case of wider spread outage. - Ensure that the deployment pipelines use immutable images for all stages https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/1800 - Prepare GCP backend bucket and a LB in front to ensure that we can recover quicker from a CDN outage https://gitlab.com/gitlab-com/gl-infra/production/-/issues/4837#note_595356824 - Consider adding redundant CDNs. - Prepare a fire-drill for cases where regular workflows are impacted (such as gitlab.com being down, or when pipelines are unable to run due to external factors). ---- **Note:** In some cases we need to redact information from public view. We only do this in a limited number of documented cases. This might include the summary, timeline or any other bits of information, laid out in out [handbook page](https://about.gitlab.com/handbook/communication/#not-public). Any of this confidential data will be in a linked issue, only visible internally. **By default, all information we can share, will be public**, in accordance to our [transparency value](https://about.gitlab.com/handbook/values/#transparency). <!-- THE BELOW IS TO BE CONDUCTED ONCE THE ABOVE INCIDENT IS MITIGATED. --> <br/> <details> <summary><i>Click to expand or collapse the Incident Review section.</i> <br/> # Incident Review </summary> <!-- The purpose of this Incident Review is to serve as a classroom to help us better understand the root causes of an incident. Treating it as a classroom allows us to create the space to let us focus on devising the mechanisms needed to prevent a similar incident from recurring in the future. A root cause can **never be a person** and this Incident Review should be written to refer to the system and the context rather than the specific actors. As placeholders for names, consider the usage of nouns like "technician", "engineer on-call", "developer", etc.. --> ## Summary <!-- _A brief summary of what happened. Try to make it as executive-friendly as possible._ _example: For a period of 19 minutes (between 2020-05-01 12:00 UTC and 2020-05-01 12:19 UTC), GitLab.com experienced a drop in traffic to the database. 507 customers saw 2,342 503 errors over this 19 minute period. The underlying cause has been determined to be a change to the PgBouncer configuration (https://gitlab.com/gitlab-com/gl-infra/production/-/issues/XXXX) which caused the total number of connections to be reduced to 50. This incident was then mitigated by rolling back this PgBouncer configuration change. --> 1. Service(s) affected: 1. Team attribution: 1. Time to detection: 1. Minutes downtime or degradation: <!-- _For calculating duration of event, use the [Platform Metrics Dashboard](https://dashboards.gitlab.net/d/general-triage/general-platform-triage?orgId=1) to look at appdex and SLO violations._ --> ## Metrics <!-- _Provide any relevant graphs that could help understand the impact of the incident and its dynamics._ --> ## Customer Impact 1. **Who was impacted by this incident? (i.e. external customers, internal customers)** 1. ... 2. **What was the customer experience during the incident? (i.e. preventing them from doing X, incorrect display of Y, ...)** 1. ... 3. **How many customers were affected?** 1. ... 4. **If a precise customer impact number is unknown, what is the estimated impact (number and ratio of failed requests, amount of traffic drop, ...)?** 1. ... ## What were the root causes? - ... ## Incident Response Analysis 1. **How was the incident detected?** 1. ... 1. **How could detection time be improved?** 1. ... 1. **How was the root cause diagnosed?** 1. ... 1. **How could time to diagnosis be improved?** 1. ... 1. **How did we reach the point where we knew how to mitigate the impact?** 1. ... 1. **How could time to mitigation be improved?** 1. ... 1. **What went well?** 1. ... ## Post Incident Analysis 1. **Did we have other events in the past with the same root cause?** 1. ... 1. **Do we have existing backlog items that would've prevented or greatly reduced the impact of this incident?** 1. ... 1. **Was this incident triggered by a change (deployment of code or change to infrastructure)? If yes, link the issue.** 1. ... ## Lessons Learned <!-- _Be explicit about what lessons we learned and should carry forward. These usually inform what our corrective actions should be._ _example:_ 1. The results of refactoring activities around our integration tests should be reviewed. (i.e we had 619 tests before refactor but 618 after.) 2. Our tooling to dedupe alarms should have integration tests to ensure it works against existing and newly added alarms. --> - ... ## Guidelines * [Blameless RCA Guideline](https://about.gitlab.com/handbook/customer-success/professional-services-engineering/workflows/internal/root-cause-analysis.html#meeting-purpose) ## Resources 1. If the **Situation Zoom room** was utilised, recording will be automatically uploaded to [Incident room Google Drive folder](https://drive.google.com/drive/folders/1wtGTU10-sybbCv1LiHIj2AFEbxizlcks) (private) </details>