Skip to content

Treat API requests from the frontend as web traffic in the rate limiter (second attempt)

Markus Koller requested to merge 344807-frontend-api-requests-rate-limit-take2 into master

What does this MR do and why?

This is a new version of !76965 (merged) which had to be reverted due to causing problems with QA tests on staging (gitlab-com/gl-infra/production#6108 (closed)).

The only change is that this is now behind a feature flag, and disabled by default. We'll look into rolling this out in #350623 (closed), after we've found a solution for https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/14200.

Original description:

This will allow us to impose stricter rate limits for general API traffic, without affecting interactive API requests made by the frontend during normal GitLab usage.

The frontend requests are identified by the inclusion of a CSRF token in the headers.

Other rate limits that only affect a subset of API requests (e.g. the Files and Packages APIs, or protected paths) still take precedence, i.e. requests for these paths will always be matched even if they include a CSRF token.

Issue: #344807 (closed), follow-up to #335300 (closed)

How to set up and validate locally

  1. Enable rate limiting with a low threshold: 
    update application_settings set throttle_unauthenticated_enabled = true, throttle_unauthenticated_api_enabled = true, throttle_unauthenticated_requests_per_period = 10, throttle_unauthenticated_api_requests_per_period = 10;
  2. Apply settings with gdk restart rails-web (you can also reset the rate limit state at any time with gdk restart redis)
  3. Without logging in, visit a page which makes API requests, e.g. http://localhost:3000/Commit451/lab-coat/-/issues/1
    • Resend one of the requests until you get a 429 response with a Ratelimit-Name: throttle_unauthenticated_web header: image
  4. Send API requests without a CSRF token, using e.g. curl -v http://localhost:3000/api/v4/groups
    • Repeat until you get a 429 response with a Ratelimit-Name: throttle_unauthenticated_api header
  5. Reset rate limiting settings: 
    update application_settings set throttle_unauthenticated_enabled = false, throttle_unauthenticated_api_enabled = false, throttle_unauthenticated_requests_per_period = 3600, throttle_unauthenticated_api_requests_per_period = 3600;

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #344807 (closed)

Edited by Markus Koller

Merge request reports