<!-- CONFIDENTIAL ISSUES: Try to keep the primary incident public. If necessary use a vague title. Any confidential information (summary, timeline, findings) should be contained in a linked, confidential issue. --> ## Current Status <!-- Leave a brief headline remark so that people know what's going on. It is perfectly acceptable for this to be vague while not much is known. --> More information will be added as we investigate the issue. <!-- UNCOMMENT AND USE WHEN MESSAGING CUSTOMERS Summary for CMOC notice / Exec summary: 1. Customer Impact: {+ Services Impacted +} 1. Customer Impact Duration: {+ start time UTC +} - {+ end time UTC +} ({+ duration in minutes +}) 1. Current state: See `Incident::<state>` label 1. Root cause: {+ TBD +} --> ## Timeline <!-- Try to capture in this section, among other events: - Time estimation for when the errors started - typically before the incident was declared. - When the incident was declared. - If other teams had to be engaged, when the right Subject Matter Expert (SME) - able to effectively work on the incident mitigation - was engaged. - When the CMOC sent first comms for this incident. - When the incident was mitigated. - When the incident was fully resolved. - A link to the original PagerDuty incident page, if any. - ... --> **[View recent production deployment and configuration events](https://nonprod-log.gitlab.net/goto/2f2872632ccd39c3895e11290c77c346)** (internal only) All times UTC. `2021-06-02` - `17:50` - @T4cC0re declares incident in Slack. ## Corrective Actions <!-- - List issues that have been created as corrective actions from this incident. - For each issue, include the following: - A one sentence summary of the corrective action. - <Bare Issue link> - Issue labeled as ~"corrective action". - If an incident review was completed, use Lessons Learned as a guideline for creation of Corrective Actions --> Corrective actions should be put here as soon as an incident is mitigated, ensure that all corrective actions mentioned in the notes below are included. - ... ---- **Note:** In some cases we need to redact information from public view. We only do this in a limited number of documented cases. This might include the summary, timeline or any other bits of information, laid out in out [handbook page](https://about.gitlab.com/handbook/communication/#not-public). Any of this confidential data will be in a linked issue, only visible internally. **By default, all information we can share, will be public**, in accordance to our [transparency value](https://about.gitlab.com/handbook/values/#transparency). <!-- THE BELOW IS TO BE CONDUCTED ONCE THE ABOVE INCIDENT IS MITIGATED. --> <br/> <details> <summary><i>Click to expand or collapse the Incident Review section.</i> <br/> # Incident Review ## Summary There was almost 10x spike in client connection to GitLab Pages for a single minute(from about 500 to almost 5000 RPS):  Service(s) affected : GitLab Pages Team attribution : ~"group::release" Minutes downtime or degradation : 1 minute ## Impact & Metrics Start with the following: | Question | Answer | |----------|--------| | What was the impact | Pages outage for 1 minute | | Who was impacted | All GitLab Pages users | | How did this impact customers | GitLab Pages sites weren't loading | | How many attempts made to access | - | | How many customers affected | Hard to estimate | | How many customers tried to access | Hard to estimate | Include any additional metrics that are of relevance. (I was debugging some, but not all of them are useful): * https://thanos-query.ops.gitlab.net/graph?g0.range_input=1h&g0.end_input=2021-06-02%2018%3A10&g0.max_source_resolution=0s&g0.expr=sum(rate(gitlab_pages_http_requests_total%5B1m%5D))&g0.tab=0&g1.range_input=1h&g1.end_input=2021-06-02%2018%3A10&g1.max_source_resolution=0s&g1.expr=avg(rate(gitlab_pages_http_request_duration_seconds_sum%5B5m%5D))&g1.tab=0&g2.range_input=1h&g2.end_input=2021-06-02%2018%3A10&g2.max_source_resolution=0s&g2.expr=avg(rate(gitlab_pages_domains_source_api_call_duration_sum%5B5m%5D))&g2.tab=0&g3.range_input=1h&g3.end_input=2021-06-02%2018%3A10&g3.max_source_resolution=0s&g3.expr=sum(rate(gitlab_pages_domains_source_api_requests_total%5B5m%5D))&g3.tab=0&g4.range_input=1h&g4.end_input=2021-06-02%2018%3A10&g4.max_source_resolution=0s&g4.expr=sum(rate(gitlab_pages_domains_source_api_call_duration_bucket%7Ble%3D%220.1%22%7D%5B5m%5D))%20%2F%20sum(rate(gitlab_pages_domains_source_api_call_duration_count%5B5m%5D))&g4.tab=0&g5.range_input=1h&g5.end_input=2021-06-02%2018%3A10&g5.max_source_resolution=0s&g5.expr=sum(rate(gitlab_pages_domains_source_api_call_duration_count%5B5m%5D))&g5.tab=0&g6.range_input=1h&g6.end_input=2021-06-02%2018%3A10&g6.max_source_resolution=0s&g6.expr=avg(rate(gitlab_pages_domains_source_api_trace_duration_sum%7Brequest_stage%3D%22httptrace.ClientTrace.GotFirstResponseByte%22%7D%5B5m%5D))&g6.tab=0&g7.range_input=1h&g7.end_input=2021-06-02%2018%3A10&g7.max_source_resolution=0s&g7.expr=avg%20by%20(request_stage)%20(rate(gitlab_pages_domains_source_api_trace_duration_sum%7B%7D%5B5m%5D))&g7.tab=0&g8.range_input=1h&g8.end_input=2021-06-02%2018%3A10&g8.max_source_resolution=0s&g8.expr=sum(rate(haproxy_backend_sessions_total%7Bfqdn%3D\~%22fe-pages-.\*%22%7D%5B1m%5D))&g8.tab=0 * https://dashboards.gitlab.net/d/api-rails-controller/api-rails-controller?orgId=1&var-PROMETHEUS_DS=Global&var-environment=gprd&var-stage=main&var-controller=Grape&var-action=GET%20%2Fapi%2Finternal%2Fpages&from=1622653800000&to=1622657400000 Hide charts (at some point I was thinking that API can be a problem, but it looks like API requests were slower only because of CPU bottleneck on the pages servers side) ## Detection & Response Start with the following: | Question | Answer | |----------|--------| | When was the incident detected? | 2021-06-02 17:50 UTC | | How was the incident detected? | Alert | | Did alarming work as expected? | Yes | | How long did it take from the start of the incident to its detection? | 1 minute | | How long did it take from detection to remediation? | 1 minute (traffic was gone by itself) | | What steps were taken to remediate? | nothing | | Were there any issues with the response? | no | ## Timeline 2021-06-02 * 17:50 UTC - traffic came * 17:51 UTC - traffic gone ## Root Cause Analysis ### Example of the usage of "5 whys" The vehicle will not start. (the problem) 1. Why? - GitLab Pages daemons were overwhelmed with traffic 1. Why? - I actually can only speculate on everything else because logs are already gone and we don't see which site was accessed. 1. Why? - 2 weeks have passed, and we store logs only for this long, and I didn't get around to look into this incident on time 1. Why? - ... 1. Why? - ... ## What went well Start with the following: * Alerting worked as expected * Pages servers were easily able to recover once additional traffic was gone ## What can be improved Start with the following: * I probably shouldn't have postponed the RCA for so long * Is there anything that could have been done to improve the detection or time to detection? - yep, it was fine * Is there anything that could have been done to improve the response or time to response? - nope, in that case response was quick, and no action was needed * Is there an existing issue that would have either prevented this incident or reduced the impact? - https://gitlab.com/gitlab-org/gitlab/-/issues/323870 * Did we have any indication or beforehand knowledge that this incident might take place? - no ## Corrective actions * List issues that have been created as corrective actions from this incident. * For each issue, include the following: * Issue labeled as ~"corrective action". * Include an estimated date of completion of the corrective action. * Include the named individual who owns the delivery of the corrective action.