## Summary Roll out [the feature](https://gitlab.com/gitlab-org/gitlab/-/work_items/625211) currently behind the `group_sync_single_authorized_projects_refresh` feature flag. - DRI: @imand3r - Team Slack channel: `#g_sscs_authorization` > [!note] > Process and guidance live in the docs — this issue is just the commands and a place to track the rollout. > "Rolling out" means incrementally enabling the flag on GitLab.com to validate stability — it is not the same as releasing the feature, which happens when the flag is removed. > [Feature flag controls](https://docs.gitlab.com/development/feature_flags/controls/) · [Feature flag lifecycle](https://handbook.gitlab.com/handbook/product-development/how-we-work/product-development-flow/feature-flag-lifecycle/#feature-flag-lifecycle) ## What could go wrong? - **Main risk:** a membership removal made by the sync is not followed by an access recalculation, so someone keeps project access they should have lost. Specs cover failures partway through, transactions and rollback, but watch production anyway. - **Watch the health dashboard:** [project authorization refresh health](https://log.gprd.gitlab.net/app/dashboards#/view/9e05a083-a7d6-44da-917a-2deb1d34e97e). A rise in access *removed* by the safety net would suggest a missed recalculation. - **Watch job counts:** `AuthorizedProjectsWorker` vs `AuthorizedProjectUpdate::UserRefreshWithLowUrgencyWorker` jobs where `json.meta.caller_id` is `GroupSamlGroupSyncWorker`. The low-urgency worker should rise and the high-urgency one should drop. Baseline and exact query: https://gitlab.com/gitlab-org/gitlab/-/work_items/625211 - **User-visible change:** project access granted by a SAML group sync can take about a minute longer to appear. - **Rollback:** instant. Disable the flag and behavior returns to one high-priority recalculation per membership change. ## Events - [Exceptions with group_sync_single_authorized_projects_refresh:1](https://log.gprd.gitlab.net/app/discover#/?_g=(time:(from:now-1d,to:now))&_a=(index:'7092c4e2-4eb5-46f2-8305-a7da2edad090',query:(language:kuery,query:'json.exception.feature_flag_states.keyword:%22group_sync_single_authorized_projects_refresh:1%22'),columns:!(json.exception.class,json.exception.message,json.extra.workflow_id,json.meta.caller_id),sort:!(!(json.time,desc)))) - [Events with group_sync_single_authorized_projects_refresh:1](https://log.gprd.gitlab.net/app/discover#/?_g=(time:(from:now-1d,to:now))&_a=(index:'7092c4e2-4eb5-46f2-8305-a7da2edad090',query:(language:kuery,query:'json.feature_flag_states:%22group_sync_single_authorized_projects_refresh:1%22'),columns:!(json.meta.caller_id,json.feature_flag_states),sort:!(!(json.time,desc)))) - [Error rate and other graphs by modifying the examples in the Visualization Library](https://log.gprd.gitlab.net/app/visualize#/?_g=h@358d019&s=FF%20Observability) Feature Flag events are only logged by default for feature flags marked for the current or future milestones. To enable while the feature flag is active, see https://docs.gitlab.com/development/feature_flags/#logging ## Rollout Run all production `/chatops` in [`#production`](https://gitlab.slack.com/archives/C101F3796) and cross-post the results to `#g_sscs_authorization`. Background: [incremental rollout process](https://docs.gitlab.com/development/feature_flags/controls/#process), [feature actors](https://docs.gitlab.com/development/feature_flags/#feature-actors). **Non-production** ``` /chatops gitlab run feature set group_sync_single_authorized_projects_refresh 50 --actors --dev --pre --staging --staging-ref /chatops gitlab run feature set group_sync_single_authorized_projects_refresh true --dev --pre --staging --staging-ref ``` **Production** — percentage rollout (wait ≥15 min between steps, watch dashboards): ``` /chatops gitlab run feature set group_sync_single_authorized_projects_refresh <percentage> --actors ``` Or target specific actors instead: ``` /chatops gitlab run feature set --project=gitlab-org/gitlab,gitlab-org/gitlab-foss group_sync_single_authorized_projects_refresh true /chatops gitlab run feature set --group=gitlab-org,gitlab-com group_sync_single_authorized_projects_refresh true /chatops gitlab run feature set --user=imand3r group_sync_single_authorized_projects_refresh true ``` ## Before global rollout Confirm the relevant gotchas before going to 100% — see [enabling a feature for GitLab.com](https://docs.gitlab.com/development/feature_flags/controls/#enabling-a-feature-for-gitlabcom): - [Docs + version history](https://docs.gitlab.com/development/documentation/feature_flags/) updated - [Breaking changes](https://docs.gitlab.com/development/documentation/release_notes/#deprecations-removals-and-breaking-changes) announced, if any - [Change management issue](https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/change-management/#feature-flags-and-the-change-management-process) opened, if required - [External API consumers](https://docs.gitlab.com/development/feature_flags/#do-not-use-feature-flags-in-external-api-consumers) handled with a fail-open mechanism, if applicable ## Cleanup Remove the flag once [deemed stable](https://handbook.gitlab.com/handbook/product-development/how-we-work/product-development-flow/feature-flag-lifecycle/#feature-flag-lifecycle) — see [cleaning up](https://docs.gitlab.com/development/feature_flags/controls/#cleaning-up). Track it here, or open a follow-up [Feature Flag Cleanup issue](https://gitlab.com/gitlab-org/gitlab/-/work_items/new?description_template=Feature%20Flag%20Cleanup). Remove the flag and its YAML definition from the codebase, then: ``` /chatops gitlab run release check <merge-request-url> <milestone> /chatops gitlab run feature delete group_sync_single_authorized_projects_refresh --dev --pre --staging --staging-ref --production ``` ## Rollback ``` /chatops gitlab run feature set group_sync_single_authorized_projects_refresh false # production /chatops gitlab run feature set group_sync_single_authorized_projects_refresh false --dev --pre --staging --staging-ref # non-production /chatops gitlab run feature delete group_sync_single_authorized_projects_refresh --dev --pre --staging --staging-ref --production # remove entirely ```