Add per-resource cost headers for rate limiting
What does this MR do and why?
During DDoS incidents, Gitaly nodes experience resource pressure from high-cost Git operations. Currently, there is no mechanism to communicate the cost of these operations to upstream rate limiters.
This MR adds two response headers to every Rails response:
x-gitlab-score-gitaly- the accumulated Gitaly RPC cost for the requestx-gitlab-namespace- the root namespace
Cloudflare's complexity-based rate limiting uses these headers to accumulate a score per namespace within a time window. When the total score exceeds the configured threshold, subsequent requests are blocked at the edge before they reach Rails or Gitaly.
The Cloudflare rule configuration would look like:
When incoming requests match:
<request-pattern>
With the same characteristics:
Header value of > x-gitlab-namespace
When rate exceeds: Complexity based
Score per period: 1000
Period: 1 minute
Response header name: x-gitlab-score-gitaly
Choose action: Block
Duration: 10 minutesCloudflare maintains a separate score counter per unique x-gitlab-namespace value. One namespace hitting the limit does not affect others. The budget threshold is configured in Cloudflare's rate limiting rules.
In this MR I have not considered tier-based scoring. The requests are spread across multiple legacy tiers (free, premium, ultimate, gold, silver, bronze, opensource, trial variants), and resolving the plan for each request would also require an additional database query.
https://log.gprd.gitlab.net/app/r/s/Oe3Kc
Closes #595523
References
- Discussion: https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/28552#note_3185494237
- Gitaly MR - gitaly!8661 (merged)
- Cloudflare complexity-based rate limiting: https://developers.cloudflare.com/waf/rate-limiting-rules/request-rate/#complexity-based-rate-limiting
Screenshots or screen recordings
| Before | After |
|---|---|
How to set up and validate locally
The Gitaly side is already merged. After updating your GDK to pick up the new Gitaly binary:
gdk restart gitaly rails-webRun any request that hits Gitaly and inspect the response headers:
curl -sI --unix-socket ~/gitlab-development-kit/gitlab.socket \
"http://localhost/gitlab-org/gitlab-test/-/blob/master/README.md" \
| grep -E "x-gitlab-score-gitaly|x-gitlab-namespace"Expected output:
x-gitlab-score-gitaly: 18
x-gitlab-namespace: gitlab-orgMR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.
Related to #595523
