[FF] oauth_lookups_use_load_balancer_sticking -- Look OAuth secrets up through a caught-up replica

Summary

Roll out the lookup side of !257390 (merged) behind the oauth_lookups_use_load_balancer_sticking feature flag (tracking issue: #579378).

With this flag on, every lookup of an OAuth access token, refresh token, or authorization code through the Doorkeeper models first asks the database load balancer for a replica that has caught up to the write location recorded when the secret was issued, or falls back to the primary. The write side that records that location is a separate flag, stick_new_oauth_tokens_to_primary (#632580). Roll that one out first: without it this flag finds no recorded location and changes nothing except the extra Redis GET.

Note

Process and guidance live in the docs — this issue is just the commands and a place to track the rollout. "Rolling out" means incrementally enabling the flag on GitLab.com to validate stability — it is not the same as releasing the feature, which happens when the flag is removed. Feature flag controls · Feature flag lifecycle

What could go wrong?

  • Blast radius: every lookup of an OAuth secret on GitLab.com. That covers Git over HTTP, container registry login, LFS, package registry auth, OpenID Connect userinfo and introspection, revoke, the refresh grant, and the authorization code exchange. API requests already honour the recorded location through Gitlab::Auth::AuthFinders#find_oauth_access_token, with or without this flag.
  • Extra load: one Redis GET on the database-load-balancing Redis per lookup of a secret shaped like a Doorkeeper secret (64 hex characters). Personal access tokens, deploy tokens and Git HTTP passwords are skipped by shape. While a recorded location exists (30 seconds after issue), one pg_last_wal_replay_lsn read per replica.
  • Data loss: none. Nothing is written.
  • What a failure looks like: nothing visible. A Redis error during the lookup is rescued, reported to Sentry, and the lookup proceeds as a normal read on whichever replica the request already has.
  • What to watch: Sentry exceptions from Doorkeeper::Concerns::LoadBalancerSticking (extra namespace: oauth_token / oauth_grant), Redis latency for the database-load-balancing instance, and the Git over HTTP / registry auth error rates.
  • Rollback: turn the flag off. Lookups go back to a plain replica read.

Events

Feature Flag events are only logged by default for feature flags marked for the current or future milestones. To enable while the feature flag is active, see https://docs.gitlab.com/development/feature_flags/#logging

Rollout

Run all production /chatops in #production and cross-post the results to #g_sscs_authentication. Background: incremental rollout process, feature actors.

Start only once stick_new_oauth_tokens_to_primary is fully enabled on the same environment.

Non-production

/chatops gitlab run feature set oauth_lookups_use_load_balancer_sticking 50 --actors --dev --pre --staging --staging-ref
/chatops gitlab run feature set oauth_lookups_use_load_balancer_sticking true --dev --pre --staging --staging-ref

Production — percentage rollout (wait ≥15 min between steps, watch dashboards):

/chatops gitlab run feature set oauth_lookups_use_load_balancer_sticking <percentage> --actors

The actor is Feature.current_request, so a percentage is a percentage of lookups, chosen per request; --project, --group and --user targeting does not apply.

Before global rollout

Confirm the relevant gotchas before going to 100% — see enabling a feature for GitLab.com:

Cleanup

Remove the flag once deemed stable — see cleaning up. Track it here, or open a follow-up Feature Flag Cleanup issue. Remove the flag and its YAML definition from the codebase. When this flag is removed, also delete the explicit read stick in Gitlab::Auth::AuthFinders#find_oauth_access_token, which the model lookup then duplicates, and restore spec/lib/gitlab/auth/auth_finders_spec.rb to expect a single find_caught_up_replica call. Then:

/chatops gitlab run release check https://gitlab.com/gitlab-org/gitlab/-/merge_requests/257390 19.5
/chatops gitlab run feature delete oauth_lookups_use_load_balancer_sticking --dev --pre --staging --staging-ref --production

Rollback

/chatops gitlab run feature set oauth_lookups_use_load_balancer_sticking false                                         # production
/chatops gitlab run feature set oauth_lookups_use_load_balancer_sticking false --dev --pre --staging --staging-ref     # non-production
/chatops gitlab run feature delete oauth_lookups_use_load_balancer_sticking --dev --pre --staging --staging-ref --production  # remove entirely