## Summary This issue is to roll out the [`/api/v4/token_exchange` endpoint](https://gitlab.com/groups/gitlab-org/-/epics/22052) on production, currently behind the `gate_token_exchange_endpoint` feature flag (introduced in !236798). The endpoint issues short-lived RS256 JWTs that modular services (initially Artifact Registry) verify via OIDC discovery + JWKS against the issuing instance. See the MR description for the full token shape, accepted token types, and authorization split. - Feature flag: `gate_token_exchange_endpoint` - Type: `gitlab_com_derisk`, default `off` - Group: `group::authentication` - Introduced in: gitlab-org/gitlab!236798 - Epic: gitlab-org&22052 ## Owners - Most appropriate Slack channel to reach out to: `#g_authentication` ## Expectations ### What are we expecting to happen? - `POST /api/v4/token_exchange` becomes reachable and returns a signed JWT for the supported credential types (legacy PAT, granular PAT, project/group access tokens, OAuth bearer, CI job token). - Deploy tokens are explicitly rejected (`401`). - No impact on any other API surface — the flag only gates the new endpoint; when disabled the route returns `404`. ### What can go wrong and how would we detect it? - **Signing-key / JWKS misalignment** — verifiers fail to validate tokens. Detected via AR-side verification error rates and `5xx` on the endpoint. - **Authentication regressions on supported token types** — track `401`/`403` rates on `/api/v4/token_exchange`. - **Increased load on `CloudConnector::Keys` / `CachingKeyLoader`** — monitor signing latency and cache hit rates. - **Unexpected entitlement behaviour** — until the AR-owned `:access_artifact_registry_service` Ability lands (work-item #599081), the policy check is a `TODO` placeholder; watch for unintended access. Relevant dashboards: API endpoint dashboards on https://dashboards.gitlab.net (filter by `/api/:version/token_exchange`). ## Rollout Steps Note: Run chatops commands in the Slack channel impacted by the command. ### Rollout on non-production environments - Verify the MR is merged to `master` and deployed to non-production: `/chatops gitlab run auto_deploy status <merge-commit>` - [x] Enable the feature globally on non-production environments: `/chatops gitlab run feature set gate_token_exchange_endpoint true --dev --pre --staging --staging-ref` - [ ] Verify that token exchange works on `staging-canary` against the AR verifier. - [ ] If end-to-end tests fail, disable the flag on staging to avoid blocking deployments. ### Specific rollout on production - Ensure the feature MRs are deployed to both production and canary: `/chatops gitlab run auto_deploy status <merge-commit>` - [ ] Enable for internal users first: `/chatops gitlab run feature set --feature-group=gitlab_team_members gate_token_exchange_endpoint true` - [ ] Verify the endpoint mints valid tokens for an internal user. ### Preparation before global rollout - [ ] Confirm the milestone on this rollout issue. - [ ] Check if the change needs a [change management issue](https://about.gitlab.com/handbook/engineering/infrastructure-platforms/change-management/#feature-flags-and-the-change-management-process); cross-link if so. - [ ] Ensure DRI availability for at least 2 hours after a production flag flip; notify `@sre-oncall` if exception needed. - [ ] Ensure documentation exists and the [version history text](https://docs.gitlab.com/development/documentation/feature_flags/#add-history-text) is updated. - [ ] Notify `#support_gitlab-com` and the team channel. ### Global rollout on production - [ ] [Incrementally roll out](https://docs.gitlab.com/development/feature_flags/controls/#process) the feature on production. - Example: `/chatops gitlab run feature set gate_token_exchange_endpoint <percentage> --actors` - Wait at least 15 minutes between steps and monitor https://dashboards.gitlab.net. - [ ] After 100% enabled, wait at least one day before releasing the feature. ### Release the feature - [ ] Create an MR removing the `gate_token_exchange_endpoint` flag: - Remove all references to the flag from the codebase. - Remove the YAML definition. - [ ] Ensure the cleanup MR is in the release package: `/chatops gitlab run release check <mr-url> <milestone>` - [ ] After the cleanup MR is deployed to production, clean up the flag: `/chatops gitlab run feature delete gate_token_exchange_endpoint --dev --pre --staging --staging-ref --production` - [ ] Close this rollout issue. ## Rollback Steps - [ ] Disable on production: ``` /chatops gitlab run feature set gate_token_exchange_endpoint false ``` - [ ] Disable on non-production: ``` /chatops gitlab run feature set gate_token_exchange_endpoint false --dev --pre --staging --staging-ref ``` - [ ] Delete from all environments: ``` /chatops gitlab run feature delete gate_token_exchange_endpoint --dev --pre --staging --staging-ref --production ```