## Summary Roll out [the feature](https://gitlab.com/gitlab-org/gitlab/-/work_items/628868) currently behind the `approval_rules_finalize_bulk_copy_approvers` feature flag. - Introduced by: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/255631 - DRI: @marc_shaw - Team Slack channel: `#g_code_review` > [!note] > Process and guidance live in the docs - this issue is just the commands and a place to track the rollout. > "Rolling out" means incrementally enabling the flag on GitLab.com to validate stability - it is not the same as releasing the feature, which happens when the flag is removed. > [Feature flag controls](https://docs.gitlab.com/development/feature_flags/controls/) · [Feature flag lifecycle](https://handbook.gitlab.com/handbook/product-development/how-we-work/product-development-flow/feature-flag-lifecycle/#feature-flag-lifecycle) ## What could go wrong? Scope: `ApprovalRules::FinalizeService#copy_project_rules`, run by `MergeWorker` after a merge on EE when the project has "Prevent editing approval rules in merge requests" enabled. With the flag on, the copied rule is saved first and its approvers are inserted with one `insert_all` per rule instead of validating and inserting every approver user one at a time. Blast radius: approvers recorded on merged merge requests in projects with that setting. The insert is idempotent on `index_approval_merge_request_rules_users_1` and sets `project_id` on every row. If it raises, the finalize transaction rolls back as before; the merge itself is already recorded. No data-loss risk, no schema change. Watch: `MergeWorker` on the [Sidekiq worker detail dashboard](https://dashboards.gitlab.net/d/sidekiq-worker-detail/sidekiq3a-worker-detail?orgId=1&var-worker=MergeWorker) and the `PatroniLongRunningTransactionDetected` alerts for `MergeWorker` referenced in the feature issue. Spot check a merged MR in such a project: the copied regular rules must list the project rule approvers, including group members. ## Events <!-- Replace feature flag name in the link--> - [Exceptions with approval_rules_finalize_bulk_copy_approvers:1](https://log.gprd.gitlab.net/app/discover#/?_g=(time:(from:now-1d,to:now))&_a=(index:'7092c4e2-4eb5-46f2-8305-a7da2edad090',query:(language:kuery,query:'json.exception.feature_flag_states.keyword:%22approval_rules_finalize_bulk_copy_approvers:1%22'),columns:!(json.exception.class,json.exception.message,json.extra.workflow_id,json.meta.caller_id),sort:!(!(json.time,desc)))) - [Events with approval_rules_finalize_bulk_copy_approvers:1](https://log.gprd.gitlab.net/app/discover#/?_g=(time:(from:now-1d,to:now))&_a=(index:'7092c4e2-4eb5-46f2-8305-a7da2edad090',query:(language:kuery,query:'json.feature_flag_states:%22approval_rules_finalize_bulk_copy_approvers:1%22'),columns:!(json.meta.caller_id,json.feature_flag_states),sort:!(!(json.time,desc)))) - [Error rate and other graphs by modifying the examples in the Visualization Library](https://log.gprd.gitlab.net/app/visualize#/?_g=h@358d019&s=FF%20Observability) Feature Flag events are only logged by default for feature flags marked for the current or future milestones. To enable while the feature flag is active, see https://docs.gitlab.com/development/feature_flags/#logging ## Rollout Run all production `/chatops` in [`#production`](https://gitlab.slack.com/archives/C101F3796) and cross-post the results to `#g_code_review`. Background: [incremental rollout process](https://docs.gitlab.com/development/feature_flags/controls/#process), [feature actors](https://docs.gitlab.com/development/feature_flags/#feature-actors). **Non-production** ``` /chatops gitlab run feature set approval_rules_finalize_bulk_copy_approvers 50 --actors --dev --pre --staging --staging-ref /chatops gitlab run feature set approval_rules_finalize_bulk_copy_approvers true --dev --pre --staging --staging-ref ``` **Production** - percentage rollout (wait ≥15 min between steps, watch dashboards): ``` /chatops gitlab run feature set approval_rules_finalize_bulk_copy_approvers <percentage> --actors ``` Or target specific actors instead: ``` /chatops gitlab run feature set --project=gitlab-org/gitlab,gitlab-org/gitlab-foss approval_rules_finalize_bulk_copy_approvers true /chatops gitlab run feature set --group=gitlab-org,gitlab-com approval_rules_finalize_bulk_copy_approvers true /chatops gitlab run feature set --user=marc_shaw approval_rules_finalize_bulk_copy_approvers true ``` ## Before global rollout Confirm the relevant gotchas before going to 100% - see [enabling a feature for GitLab.com](https://docs.gitlab.com/development/feature_flags/controls/#enabling-a-feature-for-gitlabcom): - [Docs + version history](https://docs.gitlab.com/development/documentation/feature_flags/) updated - [Breaking changes](https://docs.gitlab.com/development/documentation/release_notes/#deprecations-removals-and-breaking-changes) announced, if any - [Change management issue](https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/change-management/#feature-flags-and-the-change-management-process) opened, if required - [External API consumers](https://docs.gitlab.com/development/feature_flags/#do-not-use-feature-flags-in-external-api-consumers) handled with a fail-open mechanism, if applicable ## Cleanup Remove the flag once [deemed stable](https://handbook.gitlab.com/handbook/product-development/how-we-work/product-development-flow/feature-flag-lifecycle/#feature-flag-lifecycle) - see [cleaning up](https://docs.gitlab.com/development/feature_flags/controls/#cleaning-up). Track it here, or open a follow-up [Feature Flag Cleanup issue](https://gitlab.com/gitlab-org/gitlab/-/work_items/new?description_template=Feature%20Flag%20Cleanup). Remove the flag and its YAML definition from the codebase, then: ``` /chatops gitlab run release check <merge-request-url> <milestone> /chatops gitlab run feature delete approval_rules_finalize_bulk_copy_approvers --dev --pre --staging --staging-ref --production ``` ## Rollback ``` /chatops gitlab run feature set approval_rules_finalize_bulk_copy_approvers false # production /chatops gitlab run feature set approval_rules_finalize_bulk_copy_approvers false --dev --pre --staging --staging-ref # non-production /chatops gitlab run feature delete approval_rules_finalize_bulk_copy_approvers --dev --pre --staging --staging-ref --production # remove entirely ```