Enable Rack::Attack rate limiting for authenticated & unauthenticated requests
Enable the RackAttack rate-limiting at https://gitlab.com/admin/application_settings/network, in a carefully controlled and monitored process (a full Change Issue will be required, particularly to be really clear on how to monitor the behavior, and to have prepared responses to various eventualities). Will be done in dry-run mode first, to validate and perhaps tune the limits, before enabling it for real.
Spun off from #625 (closed).
Settings
For the record, these are the values we set on gitlab.com and when we set them (to be filled in as we go through this process). Each cell contains <count>/<period> with (dry) when its in dry-run mode
| Date | Unauth | Auth API | Auth Web | Comment |
|---|---|---|---|---|
| 2020-11-16 | 3600/h | 7200/h | 7200/h | Not enabled yet at all, even dry run |
| 2020-11-20 | 1500/m (dry) | 1000/m (dry) | 2000/m (dry) | Note the numbers were inadvertently transposed for the two auth options |
| 2020-11-27 | 1500/m (dry) | 2000/m (dry) | 1000/m (dry) | Fixed the transposed limits to the intended values |
| 2020-12-22 | 500/m (dry) | 2000/m (dry) | 1000/m (dry) | Dropped the unauthenticated limit as proposed, to see what happens, at ~ 02:12 UTC |
| 2021-01-18 | 500/m | 2000/m | 1000/m | Enabled in production#3324 |
Work to be done on this issue
Prepare to turn on enforcing mode by making sure that:
-
Rate-limiting enabled in dry-run mode -
Data gathered on current breaches -
Determine if other customers need to be contacted (handled on epic) -
Complete other issues on this epic before enabling
Also a private snippet https://gitlab.com/gitlab-com/gl-infra/scalability/-/snippets/2045448 with exclusions/commentary on the log analysis (private, because it contains usernames and IP addresses)