Cloudflare: Move HAProxy-based rate limiting to Cloudflare

Summary

We should move the rate-Limiting we currently have in HAProxy to Cloudflare.

Because of the way we do rate-limiting in HAProxy, we require a workaround, which terminates the connection to Cloudflare after every request.

This is because we set connection registers to the source IP of the request, which we extract from the header passed by Cloudflare.
The problem here is, that Cloudflare will multiplex different requests from different source IPs over the same connection. Leading to the first request on the connection setting the IP, which is then carried forward for subsequent requests on the same connection.

This workaround causes us to do more TLS handshakes than otherwise necessary, and leads to lower utilization of our capabilities, which are wasted by TLS handshakes.

Moving the rate limit to Cloudflare would mean, that we can remove the rate-limiting in HAProxy, which in turn allows us to remove the workaround with terminating the connection after every request.

Related Incident(s)

Originating issue(s): https://gitlab.com/gitlab-com/gl-infra/production/-/issues/2727

Desired Outcome/Acceptance criteria

The ultimate goal is to be able to disable the registry and API rate limiting in HAProxy.

One measure of success would be that these two sets of information is similar:

Important!: Disabling rate-limiting in HAProxy is out of scope (handled in https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/9710). This is just to transfer the rate-limits from the ACLs to Cloudflare to unblock that issue.

Associated Services

Corrective Action Issue Checklist

  • link the incident(s) this corrective action arose out of
  • give context for what problem this corrective action is trying to prevent from re-occurring
  • assign a severity label (this is the highest sev of related incidents, defaults to 'severity::4')
  • assign a priority (this will default to 'priority::4')
Edited by Hendrik Meyer (xLabber)