Count expired OAuth tokens per token in rate limiter

What does this MR do and why?

On GitLab.com, the labkit rack rate limiter cannot find a requester for a request with an expired OAuth token. Gitlab::Auth::RequestAuthenticator#find_sessionless_user rescues the auth error and returns nil. The middleware Gitlab::Middleware::LabkitRackRateLimit then counts the request under the per-IP rule unauthenticated_traffic_per_ip. This rule allows 60 requests per hour per IP.

When the IP is over the limit, the middleware returns 429 before the API can return 401. A client that refreshes its OAuth token on a 401 never gets the 401, so it never refreshes. The client stays blocked until the window ends or the user reconnects. One stale client also blocks other tools and users on the same IP.

This MR changes the following:

  • Add the fact expired_oauth_token_id to ee/lib/ee/gitlab/rack_attack/labkit_rate_limit/classified_request.rb. The classifier reads it from the auth failure that Gitlab::Auth::AuthFinders records in Gitlab::ApplicationContext. The fact is set when auth_fail_reason is token_expired and auth_fail_token_id starts with OauthAccessToken/.
  • Set the fact only on paths that answer an expired token with a 401: paths that start with /api/, and the OAuth token check paths (oauth_token_check_path?). Feed paths (.atom, .ics) also check the token, but they serve the page anonymously. api_request? is not used, because it matches /oauth/ anywhere, including a group or project named oauth.
  • Add the rule pair expired_oauth_token_traffic_per_token and expired_oauth_token_traffic_per_token_log. They allow 60 requests per hour for each expired token. They use the same paths and the same info flag as the per-IP rule, and split on expired_oauth_token_enforced.
  • Make the four per-IP unauthenticated rules skip a request only when the new fact expired_oauth_token_enforced is true. It is true only for an expired OAuth token with both flags and the unauthenticated enforce state on.
  • Add a per-IP cap, expired_oauth_token_traffic_per_ip and its _log twin, 1,200 per hour per IP. It comes after the per-token rule. Labkit stops the rule walk at the first over-limit :limit rule, so a token past its own 60 does not debit the cap. The busiest real IP in the sample had 17 expired tokens, so real traffic stays under about 1,020 an hour.
  • Add two gitlab_com_derisk feature flags, both off by default. rate_limiter_expired_oauth_token_per_token starts an observe stage: the two _log rules count expired-token requests, and the per-IP unauthenticated rules still count them as today. rate_limiter_expired_oauth_token_per_token_enforce starts the enforce stage: the per-token and per-IP :limit rules count and block, and the per-IP unauthenticated rules skip the request. The enforce stage also needs rate_limiter_unauthenticated_limits_enforce on. With only the main flag on, no client sees a change.

A request within the per-token limit now reaches the API and gets the 401.

Expired personal access tokens and all other invalid, revoked, or wrong-scope tokens stay on the per-IP rule. The new rules do not check the target namespace's plan. In the enforce stage, an expired OAuth token on an API path in a paid namespace moves from the 10,000 per hour per-IP budget (unauthenticated_paid_namespace_traffic_per_ip) to 60 per hour per token. This is intended: the request gets a 401 and no data, and the client refreshes after the first one.

No changelog entry, because the change is behind a default-off feature flag.

Why 60 requests per hour, and why only OAuth tokens

Production logs for one weekday hour (2026-10-23, 12:00 to 13:00 UTC) show 8,506 expired OAuth tokens.

Requests per token in that hour Value
Median 1
90th percentile 6
95th percentile 16
99th percentile 190

169 tokens (1.9%) went over 60. These are tools that keep sending a token that is long expired and never refresh it. Under the new rule, each of these tokens gets 60 401 responses an hour and then a 429 for that token only. Before, they used up the budget of the whole IP.

Why other tokens stay on the per-IP rule:

  • An earlier decision (https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/30053) kept invalid tokens on the per-IP rule. This keeps token scanners limited.
  • An expired OAuth token was a real token that was once valid. A scanner cannot guess one, and each token is still limited to 60 requests per hour.
  • Counting expired tokens against the per-user limit of the token owner was rejected. A person with leaked expired tokens could use up the limit of that user.

Out of scope:

  • Client fixes, for example IDE extensions that keep sending an expired token, and agent providers that retry a 429 at once.
  • A review of the dependency proxy exemption from !259996 (merged). This is a separate follow-up.

References

Screenshots or screen recordings

Not applicable. This MR has no UI change.

How to set up and validate locally

  1. Start GDK with GITLAB_SIMULATE_SAAS=1, so Gitlab::Saas.feature_available? returns true.

  2. In the Rails console, enable the flags:

    Feature.enable(:rate_limiter_unauthenticated_limits_info)
    Feature.enable(:rate_limiter_expired_oauth_token_per_token)
  3. In the Rails console, create an OAuth access token that is already expired, and copy its plaintext value:

    token = OauthAccessToken.create!(
      application: Authn::OauthApplication.first,
      resource_owner_id: User.find_by_username('root').id,
      organization: Organizations::Organization.first,
      scopes: 'api',
      expires_in: 1.hour,
      created_at: 3.hours.ago
    )
    token.plaintext_token
  4. Send a request with the expired token. Expect a 401 response:

    curl -i -H "Authorization: Bearer <token>" http://127.0.0.1:3000/api/v4/user
  5. In the Rails console, check the Redis counter. The command returns one key:

    Gitlab::Redis::RateLimiting.with { |r| r.keys('labkit:rl:{rack_request:expired_oauth_token_traffic_per_token_log:*') }

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Hardik Gala

Merge request reports

Loading
Loading