Skip to content

Allow read_api scope in rate limiter

Sean McGivern requested to merge fix-rack-attack-with-read-api-scope into master

What does this MR do?

Previously, requests using a token that only had read_api access would be treated as unauthenticated by the rate limiter. This ensures that they are (correctly) treated as authenticated.

This does point to a general problem with the rate limiting code: because it happens in a middleware before any application code, it's hard to keep in sync with the different authentication methods the application may allow. This applies to things like:

  1. How the credentials are sent (Private-Token header, Authorization header, basic auth, etc.).
  2. What credentials we accept (a personal access token, a personal access token with read_api scope, a job token, a deploy token, etc.).
  3. Which endpoints those apply to.

The last one is mostly a concern for performance: we might pre-emptively try authenticating using a method that isn't valid for a particular endpoint. From a rate limiting perspective it doesn't matter so much, as we're treating a request with valid credentials for a different endpoint as authenticated for the current endpoint, which is wrong but not a significant problem.

We're not solving the first two in this MR, just continuing to play whack-a-mole.

Testing this locally

  1. Create a token with read_api scope and put it in a GITLAB_API_TOKEN_LOCAL_READ environment variable.
  2. Enable rate limits for unauthenticated requests at a rate of 1 request per 60 seconds.
    1. Note: do this second, and undo it afterwards. Otherwise things like logging in become tricky 😃

On master, you'll see this:

$ curl -s -i -H "Private-Token: $GITLAB_API_TOKEN_LOCAL_READ" http://localhost:3000/api/v4/users/1 | head -n 1
HTTP/1.1 200 OK
$ curl -s -i -H "Private-Token: $GITLAB_API_TOKEN_LOCAL_READ" http://localhost:3000/api/v4/users/1 | head -n 1
HTTP/1.1 429 Too Many Requests

Along with an entry in log/auth.log for the second request with throttle_unauthenticated:

{"severity":"ERROR","time":"2020-12-30T13:58:43.863Z","correlation_id":"01ETSZ365Z6FTAYED0K8SDY2G1","message":"Rack_Attack","env":"throttle","remote_ip":"127.0.0.1","request_method":"GET","path":"/api/v4/users/1","matched":"throttle_unauthenticated"}

And on this branch, you'll see:

$ curl -s -i -H "Private-Token: $GITLAB_API_TOKEN_LOCAL_READ" http://localhost:3000/api/v4/users/1 | head -n 1
HTTP/1.1 200 OK
$ curl -s -i -H "Private-Token: $GITLAB_API_TOKEN_LOCAL_READ" http://localhost:3000/api/v4/users/1 | head -n 1
HTTP/1.1 200 OK

For gitlab-com/gl-infra/scalability#753 (closed).

Edited by Sean McGivern

Merge request reports