2020-05-30: A spike of requests to a single Pages site
Summary
At 14:24 UTC, we started receiving a large number of requests to a single Pages site, maxing out to 19.05K requests. The spike lasted for 2 minutes. We were alerted after the fact.
Timeline
All times UTC.
2020-05-30
- 14:24 - A spike of Pages requests is starting
- 14:26 - The spike ended
- 14:27 - EOC is pages about increased backend errors
- 14:36 - The source of the requests is identified to be CloudFlare
Incident Review
Summary
- Service(s) affected :
- Team attribution :
- Minutes downtime or degradation :
Metrics
Customer Impact
- Who was impacted by this incident? (i.e. external customers, internal customers)
- What was the customer experience during the incident? (i.e. preventing them from doing X, incorrect display of Y, ...)
- How many customers were affected?
- If a precise customer impact number is unknown, what is the estimated potential impact?
Incident Response Analysis
- How was the event detected?
- How could detection time be improved?
- How did we reach the point where we knew how to mitigate the impact?
- How could time to mitigation be improved?
Post Incident Analysis
- How was the root cause diagnosed?
- How could time to diagnosis be improved?
- Do we have an existing backlog item that would've prevented or greatly reduced the impact of this incident?
- Was this incident triggered by a change (deployment of code or change to infrastructure. If yes, have you linked the issue which represents the change?)?
Timeline
- YYYY-MM-DD XX:YY UTC: action X taken
- YYYY-MM-DD XX:YY UTC: action Y taken
5 Whys
Lessons Learned
Corrective Actions
Guidelines
Resources
- If the Situation Zoom room was utilised, recording will be automatically uploaded to Incident room Google Drive folder (private)
Activity
added ServicePages label
The response I got from CF:
Hi,
There are two circumstances where it might appear that Cloudflare is attacking your site.
- You're a Cloudflare customer for your website(s). Since CloudFlare is a reverse proxy for our customers' sites, Cloudflare IPs are going to show in your server logs until you install something on your server to restore original visitor IP, such as mod_cloudflare for Apache servers. Solutions for seeing original visitor IP for Apache, nginx and other servers and applications are listed here: https://support.cloudflare.com/hc/en-us/sections/200805497-Restoring-Visitor-IPs
- You're getting attacks from Cloudflare's IPs because they are being spoofed. Cloudflare does not send traffic over anything other than http:// (ports 80 and 443), so getting attacked by UDP requests means you are likely seeing a DNS amplification attack, see this article for more information.
I've reviewed our logs. We see the non-Cloudflare IPs making these requests and believe this is the first case.
[...]
So the spike was legit albeit excessive. I don't see any more actions to be taken here, so I'll close the issue.
mentioned in issue on-call-handovers#650 (closed)
mentioned in issue on-call-handovers#651 (closed)
mentioned in issue on-call-handovers#652 (closed)
mentioned in issue on-call-handovers#653 (closed)
mentioned in issue on-call-handovers#654 (closed)
mentioned in issue on-call-handovers#655 (closed)
mentioned in issue #2214 (closed)
This incident was closed before the IncidentReview-Completed label was applied. The issue is being reopened so that it will appear on the Production Incidents board—and can be moved through the entire incident management workflow.
Please review the Incident Workflow section on the Incident Management handbook page for more information.
/open
added auto updated label
-
This incident was closed before the IncidentReview-Completed label was applied. The issue is being reopened so that it will appear on the Production Incidents board—and can be moved through the entire incident management workflow.
Please review the Incident Workflow section on the Incident Management handbook page for more information.
/open