Allow configurable bypass of RackAttack rate limiting middleware for a specific header
Related to https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/9062
At present RackAttack is disabled on GitLab.com, and we rely on IP based rate limiting in haproxy.
This approach has numerous downsides, and these have been leading to multiple incidents recently.
In https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/9062 some effort was made to enable RackAttack, but this since GitLab's RackAttack does not allow configuration of allow-lists, we had to turn it off as it caused problems for some clients.
Proposal
Allow RackAttack to be disabled for requests with a specific header.
For example, the header could be X-GitLab-RateLimit-Bypass
. This could be enabled on GitLab.com.
X-GitLab-RateLimit-Bypass
would be filtered by haproxy, so that clients could not pass it through. For security reasons it could only be set in HAProxy (or potentially other trusted proxies in future, eg CloudFlare)
On GitLab.com, we could allow our existing allow lists to be used. When an allow list ACL matches, haproxy adds the header.
The administrator configuration would look something like this
- Administrator configures haproxy to drop all
X-GitLab-RateLimit-Bypass
headers from clients - Administrator configures haproxy to insert the
X-GitLab-RateLimit-Bypass
header into downstream requests for matching ACLS, using the existing allow-list ACLs that we already have in place. - Administrator opens GitLab Admin UI
- Administrator enables "Bypass RateLimit for Header" and fills in the
X-GitLab-RateLimit-Bypass
header name in the field.
When this is enabled, rate limiting will be applied to all traffic except for allow-list matches from customers, internal APIs etc.
This approach gives us a great deal of flexibility going forward while only requiring small changes up front.
Additionally, the allow lists can use any filtering supported by HAProxy ACLs, including IPv4 addresses, IPv6, CIDR blocks.