403s from unique IP rate limiter
Because of https://gitlab.com/gitlab-org/gitlab-ce/issues/29674, the unique IP rate limiter was still on when it should have been turned off. The settings were set as follows:
- 10 unique IPs over 3600 seconds (1 hour)
It seems, however, there is some issue with this because a number of users, including @brodock, were reporting 403 errors throughout the day.
At first, we suspected it was possible the wrong IP was being used:
gabriel: if the reverse proxy is passing the IPs using an special header like x-forwarded but rack-attack doesn’t understand it, it will flag requests coming from different “workers” as a violation.
When I checked the latest Redis state today (long after the setting was deactivated), the right IPs seem to be there. For example, in @brodock's case:
> zrange 'user_unique_ips:2293' 0 -1
1) "77.173.x.x"
It's possible something else (e.g. Rack Attack) might be playing a role here, but the Kibana logs show nothing of the sort. Also, @brodock tried using a VPN and had the same issues, suggesting the unique IP rate limiter was in action.