Container Registry /v2/ API requests are throttled under the authenticated web rate limit

Summary

/v2/ Container Registry API requests are classified as web requests and counted against the authenticated web throttle (throttle_authenticated_web), rather than a dedicated API throttle. This can cause CI/CD pipelines doing image pulls to receive unexpected 429 responses when the web rate limit is reached. That also means that Container Registry and Dependency Proxy traffic is competing with general web traffic for the rate limited requests per user.

The throttle_authenticated_web? method for Gitlab::RackAttack::Request might be where it determines whether to throttle for web requests: https://gitlab.com/gitlab-org/gitlab/-/blob/3c229dce643472146e27a1791d03cbdf3aa46140/lib/gitlab/rack_attack/request.rb#L119-124

It seems to determine whether it's a (web_request? || frontend_request?): https://gitlab.com/gitlab-org/gitlab/-/blob/3c229dce643472146e27a1791d03cbdf3aa46140/lib/gitlab/rack_attack/request.rb#L72

!api_request? && !health_check_request? && !product_analytics_collector_request?

And api_request? checks if the path matches the API_PATH_REGEX https://gitlab.com/gitlab-org/gitlab/-/blob/3c229dce643472146e27a1791d03cbdf3aa46140/lib/gitlab/rack_attack/request.rb#L40

API_PATH_REGEX = %r{^/api/|/oauth/}

And the path doesn't match this regular expression, so it's treated as a web request. This also seems to impact /api/graphql, although a lot of GraphQL requests do originate from the UI.

Steps to reproduce

  1. Set throttle_authenticated_web_enabled to true: ApplicationSetting.first.update!(throttle_authenticated_web_enabled: true)
  2. Ensure the Runner has at least 6 configured globally for concurrent
  3. Set throttle_authenticated_web_period_in_seconds to 10 and throttle_authenticated_web_requests_per_period to 20. For a self-managed instance, this is easier than manufacturing a lot of load to hit the rate limits per user
  4. Create a project, and add a Dockerfile and CI/CD YAML file as per the file definitions below
  5. Run the pipeline and refresh the page a few times while viewing the pipeline
stages:
  - build

.build-job:
  image: ${CI_DEPENDENCY_PROXY_DIRECT_GROUP_IMAGE_PREFIX}/docker:24.0.5-cli
  services:
    - name: ${CI_DEPENDENCY_PROXY_DIRECT_GROUP_IMAGE_PREFIX}/docker:24.0.5-dind
      alias: docker
      variables:
        HEALTHCHECK_TCP_PORT: "2376"
  variables:
    DOCKER_HOST: tcp://docker:2376
    DOCKER_TLS_CERTDIR: "/certs"
    DOCKER_TLS_VERIFY: 1
    DOCKER_CERT_PATH: "$DOCKER_TLS_CERTDIR/client"
  script:
    - docker login -u $CI_DEPENDENCY_PROXY_USER -p $CI_DEPENDENCY_PROXY_PASSWORD $CI_DEPENDENCY_PROXY_SERVER
    -  docker build --build-arg CI_DEPENDENCY_PROXY_DIRECT_GROUP_IMAGE_PREFIX=$CI_DEPENDENCY_PROXY_DIRECT_GROUP_IMAGE_PREFIX .

build:
  stage: build
  extends: .build-job
  parallel: 10
ARG CI_DEPENDENCY_PROXY_DIRECT_GROUP_IMAGE_PREFIX
FROM ${CI_DEPENDENCY_PROXY_DIRECT_GROUP_IMAGE_PREFIX}/alpine:latest as one
FROM ${CI_DEPENDENCY_PROXY_DIRECT_GROUP_IMAGE_PREFIX}/busybox:latest as two

Example Project

What is the current bug behavior?

Pipeline fails while executing the Docker build, the pull from Dependency Proxy gets 429 status code. Rails logs show throttle_authenticated_web and Rack_Attack

What is the expected correct behavior?

Unsure, but it's confusing that /v2/ REST API traffic is counted towards authenticated web request rate limits

Relevant logs and/or screenshots

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info 
(For installations with omnibus-gitlab package run and paste the output of:
`sudo gitlab-rake gitlab:env:info`)

(For installations from source run and paste the output of:
`sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production`)

Results of GitLab application Check

Expand for output related to the GitLab application check 
(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true)


(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true)


(we will only investigate if the tests are passing)

Possible fixes

Patch release information for backports

If the bug fix needs to be backported in a patch release to a version under the maintenance policy, please follow the steps on the patch release runbook for GitLab engineers.

Refer to the internal "Release Information" dashboard for information about the next patch release, including the targeted versions, expected release date, and current status.

High-severity bug remediation

To remediate high-severity issues requiring an internal release for single-tenant SaaS instances, refer to the internal release process for engineers.

Edited by Michael Trainor