Skip to content

Treat API requests from the frontend as web traffic in the rate limiter (third attempt 馃)

Markus Koller requested to merge 344807-frontend-api-requests-rate-limit-take3 into master

What does this MR do and why?

This is a new version of !78082 (merged) which had to be reverted due to causing a production incident (gitlab-com/gl-infra/production#6207 (closed)).

We previously reverted that MR in !76965 (merged) and put it behind a FF, but it then turned out that I introduced a bug in the original MR which was likely to be the root cause for this and previous incidents: !78082 (comment 817796863). The refactoring which caused this bug was extracted into !79528 (merged).

We'll properly test this change on staging first before rolling it out to production: #350623 (closed)

Original description:

This will allow us to impose stricter rate limits for general API traffic, without affecting interactive API requests made by the frontend during normal GitLab usage.

The frontend requests are identified by the inclusion of a CSRF token in the headers.

Other rate limits that only affect a subset of API requests (e.g. the Files and Packages APIs, or protected paths) still take precedence, i.e. requests for these paths will always be matched even if they include a CSRF token.

Issue: #344807 (closed), follow-up to #335300 (closed)

How to set up and validate locally

  1. Enable the feature flag: 
    Feature.enable(:rate_limit_frontend_requests)
  2. Enable rate limiting with a low threshold: 
    update application_settings set throttle_unauthenticated_enabled = true, throttle_unauthenticated_api_enabled = true, throttle_unauthenticated_requests_per_period = 10, throttle_unauthenticated_api_requests_per_period = 10;
  3. Apply settings with gdk restart rails-web (you can also reset the rate limit state at any time with gdk restart redis)
  4. Without logging in, visit a page which makes API requests, e.g. http://localhost:3000/Commit451/lab-coat/-/issues/1
    • Make sure the request contains a X-CSRF-Token header (all API requests made by the frontend should have one).
    • Resend one of the requests until you get a 429 response with a Ratelimit-Name: throttle_unauthenticated_web header: image
  5. Send API requests without a CSRF token, using e.g. curl -v http://localhost:3000/api/v4/groups
    • Repeat until you get a 429 response with a Ratelimit-Name: throttle_unauthenticated_api header
  6. Reset rate limiting settings: 
    update application_settings set throttle_unauthenticated_enabled = false, throttle_unauthenticated_api_enabled = false, throttle_unauthenticated_requests_per_period = 3600, throttle_unauthenticated_api_requests_per_period = 3600;

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #344807 (closed)

Edited by Markus Koller

Merge request reports