Non-IP Rate Limiting Bypass Header logic is moved out of HAProxy

Problem

In https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/issues/25939 we moved the IP Based Rate Limiting Bypass Header logic out of HAProxy and into Cloudflare, to make management of customer allow lists easier.

There are four additional non-IP based bypasses in place, that we should also move before we mark this effort as complete, to minimise confusion, and ensure all bypass management is controlled in one place.

Rule Related Issue Matcher

is_https_git

https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/691+

Path (regex)

{ path -i /jwt/auth }

https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/692+ ->

Add separate RackAttack throttle for /jwt/auth (gitlab-org/gitlab#330107)

Path

is_gitlab_com_package_registry

https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/710+

Path (regex)

is_gitlab_com_go_get

https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/710+

Query string

Related note.

Proposal

Create Cloudflare custom rules that set the X-GitLab-RateLimit-Bypass header for the following:

  • Path based: git
  • Path based: jwt/auth
  • Path: package registry
  • Query string: go get

The logic should match the existing logic in HAProxy, and we should validate that the traffic patterns for any changes in traffic volumes. Example logs for /jwt/auth

  • Remove non-IP based bypass header logic from HAProxy
Edited by Adeline Yeung