OAuth /oauth/token refresh grant returns transient invalid_grant 400 due to DB replica lag

Summary

The /oauth/token endpoint intermittently returns invalid_grant 400 errors for valid refresh tokens. The same refresh token succeeds on a subsequent request minutes later. This has been observed during OAuth permissions syncs but affects any OAuth client using the refresh token grant flow.

Root Cause

The refresh token grant flow in Doorkeeper bypasses GitLab's load balancer stickiness logic, causing token lookups to hit DB replicas that haven't yet caught up with the primary.

When a refresh token is issued, it is written to the primary DB. If the next request using that refresh token is routed to a different pod whose DB connection hits a lagging replica, Doorkeeper cannot find the hashed token and returns invalid_grant. A subsequent retry minutes later hits a caught-up replica and succeeds with the same token.

This is confirmed by production logs showing:

  • The 400 and 200 responses are returned by GitLab itself (not an intermediate proxy), as confirmed by workhorse logs
  • The two requests hit different pods on different GKE nodes
  • The request body (read_bytes) is identical in both cases, confirming the same refresh token was used
  • The response body (written_bytes) differs: 213 bytes for the 400, 259 bytes for the 200

Existing Mitigation (Not Applied Here)

GitLab already handles this for bearer token validation in lib/gitlab/auth/auth_finders.rb:

# Ensure we use correct load balancing logic in case it's a newly created token
load_balancer_stick_request(
  ::OauthAccessToken,
  :oauth_token,
  current_token,
  hash_id: true
)

However, this stickiness is not applied to the /oauth/token refresh grant flow, which goes through Doorkeeper directly and performs its own DB lookup without sticky routing.

Proposed Fix

Apply load balancer primary stickiness for the refresh token lookup in the Doorkeeper grant flow. This likely requires overriding the relevant Doorkeeper refresh token strategy to call load_balancer_stick_request before the token lookup, similar to the existing pattern in find_oauth_access_token.

Candidate locations:

  • app/controllers/oauth/tokens_controller.rb
  • A Doorkeeper initializer override in config/initializers/doorkeeper.rb

Client-Side Mitigation

Until the server-side fix is in place, OAuth clients should treat invalid_grant 400 responses as potentially transient when the token is freshly issued, and retry with short exponential backoff (capped at 2-3 attempts). The existing Bitbucket::ExponentialBackoff module (lib/bitbucket/exponential_backoff.rb) is a reusable pattern for this.

Impact

  • Any OAuth client using the refresh token grant flow against GitLab.com can encounter this
  • Particularly visible during bulk operations (e.g., permissions syncs) that issue many tokens in rapid succession across multiple pods
  • Results in failed syncs, authentication errors, and degraded user experience for integrations relying on OAuth

References

  • Existing load balancer stickiness pattern: lib/gitlab/auth/auth_finders.rb (find_oauth_access_token)
  • Doorkeeper initializer: config/initializers/doorkeeper.rb
  • Token hashing: Gitlab::DoorkeeperSecretStoring::Sha512Hash
Edited by 🤖 GitLab Bot 🤖