<!--IssueSummary start--> <details> <summary> Everyone can contribute. [Help move this issue forward](https://handbook.gitlab.com/handbook/marketing/developer-relations/contributor-success/community-contributors-workflows/#contributor-links) while earning points, leveling up and collecting rewards. </summary> - [Label this issue](https://contributors.gitlab.com/manage-issue?action=label&projectId=69721933&issueIid=148) </details> <!--IssueSummary end--> ## Summary Roll out [the feature](https://gitlab.com/gitlab-org/embody-team/experimental-observability/documentation/-/work_items/141) currently behind the `observability_per_user_bff_auth` feature flag. - DRI: @dakotadux - Team Slack channel: `#g_observability` > [!note] > Process and guidance live in the docs — this issue is just the commands and a place to track the rollout. > "Rolling out" means incrementally enabling the flag on GitLab.com to validate stability — it is not the same as releasing the feature, which happens when the flag is removed. > [Feature flag controls](https://docs.gitlab.com/development/feature_flags/controls/) · [Feature flag lifecycle](https://handbook.gitlab.com/handbook/product-development/how-we-work/product-development-flow/feature-flag-lifecycle/#feature-flag-lifecycle) ## What could go wrong? The flag gates the backend-for-frontend (BFF) broker that replaces the shared-credential SigNoz iframe login with a per-user, GitLab-brokered OIDC exchange (`Observability::O11yBffSession`). Enabling it for a group/project: - Changes how `app.vue` authenticates into the embedded SigNoz iframe (BFF POST instead of polling the existing session endpoint). - Depends on the shared SigNoz OAuth application being correctly configured via the new `signoz_oauth_application_id` instance setting -- if unset, the broker fails closed and users see the auth error state (no fallback to the old shared-credential flow while the flag is on for that actor). - Depends on infra from documentation#142 (dedicated mTLS ALB listener + client cert) for the SigNoz-side flow to actually validate the request; until that infra exists, the exchange happens without a client cert and SigNoz's own mTLS enforcement (once live) will reject it. Watch for elevated auth-error rates on the observability iframe and `O11y BFF session exchange failed` warnings in `Gitlab::AppLogger`. ## Rollout Run all production `/chatops` in [`#production`](https://gitlab.slack.com/archives/C101F3796) and cross-post the results to `#g_observability`. Background: [incremental rollout process](https://docs.gitlab.com/development/feature_flags/controls/#process), [feature actors](https://docs.gitlab.com/development/feature_flags/#feature-actors). **Non-production** ``` /chatops gitlab run feature set observability_per_user_bff_auth 50 --actors --dev --pre --staging --staging-ref /chatops gitlab run feature set observability_per_user_bff_auth true --dev --pre --staging --staging-ref ``` **Production** — percentage rollout (wait ≥15 min between steps, watch dashboards): ``` /chatops gitlab run feature set observability_per_user_bff_auth <percentage> --actors ``` Or target specific actors instead: ``` /chatops gitlab run feature set --group=<test-group> observability_per_user_bff_auth true ``` ## Before global rollout Confirm the relevant gotchas before going to 100% — see [enabling a feature for GitLab.com](https://docs.gitlab.com/development/feature_flags/controls/#enabling-a-feature-for-gitlabcom): - [ ] documentation#142 (mTLS infra) merged and verified in production - [ ] documentation#140 (provisioner OIDC seeding) merged and verified in production - [ ] Shared SigNoz OAuth application configured via the instance-level setting on GitLab.com - [ ] Docs + version history updated - [ ] No breaking changes for external API consumers (n/a -- internal iframe auth only) ## Cleanup Remove the flag once [deemed stable](https://handbook.gitlab.com/handbook/product-development/how-we-work/product-development-flow/feature-flag-lifecycle/#feature-flag-lifecycle) — see [cleaning up](https://docs.gitlab.com/development/feature_flags/controls/#cleaning-up). Remove the flag and its YAML definition from the codebase, then: ``` /chatops gitlab run release check <merge-request-url> <milestone> /chatops gitlab run feature delete observability_per_user_bff_auth --dev --pre --staging --staging-ref --production ``` ## Rollback ``` /chatops gitlab run feature set observability_per_user_bff_auth false # production /chatops gitlab run feature set observability_per_user_bff_auth false --dev --pre --staging --staging-ref # non-production /chatops gitlab run feature delete observability_per_user_bff_auth --dev --pre --staging --staging-ref --production # remove entirely ``` ## References Part of https://gitlab.com/gitlab-org/embody-team/experimental-observability/documentation/-/work_items/141 Relates to https://gitlab.com/gitlab-org/embody-team/experimental-observability/documentation/-/work_items/138