Configurable rate limits for unauthenticated requests to raw endpoint

Joe Woodwardrequested to merge
feat/unauthenticated_raw_blob_rate_limit into master

What does this MR do and why?

Configurable rate limits for unauthenticated requests to raw endpoint

Currently all traffic to the raw endpoint is throttled as a single client, scoped to project and path. So a user can make 300 requests for each raw blob in a project.

However, unauthenticated users can also submit the same amount of requests and by using more than 1 IP address they can easily get around our rate limits.

This change allows administrators to configure per project rate limits for unauthenticated requests.

Changelog: added

References

https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/28492+

Screenshots or screen recordings

image

How to set up and validate locally

  1. Go to the admin area and configure the throttles in Settings > Network > Performance optimizations
  2. Set the unauthenticated throttle to something low
  3. Trigger a bunch of curl commands to the raw endpoint (replace URL for your own)
COUNT=0; while true; do COUNT=$((COUNT + 1)); echo "Request #$COUNT"; STATUS=$(curl -s -o /dev/null -w "%{http_code}" "https://gdk.test:3443/gitlab-org/gitlab-test/-/raw/master/10m_file.text"); echo "  Response: $STATUS"; [ "$STATUS" -eq 429 ] && echo -e "\nSuccessfully triggered rate limiter" && break; done

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Merge request reports