## Summary This issue is to roll out [the feature](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/174453) on production, that is currently behind the `org_mover_extend_selective_sync_to_primary_checksumming` feature flag. The change was added to facilitate migrating orgs across cells on gitlab.com. However we weren't going to bother with the overhead to release it GA anytime soon, but instead we would use it first for our Cells use-case. After that, we could opt to release it at our convenience. When enabled, the primary (source cell) scopes its checksumming/verification work to the records that selective sync would replicate, instead of checksumming all replicables of each type. This is needed so that Org Mover / selective-sync deployments on large instances (GitLab.com) do not run instance-wide checksumming queries against massive tables. Note: This is an **ops** feature flag checked **without an actor** (`Gitlab/FeatureFlagWithoutActor` — checksumming is instance-wide). The flag is read in `Gitlab::Geo.org_mover_extend_selective_sync_to_primary_checksumming?` and applied in the `verifiables` scope (`ee/app/models/concerns/geo/replicable_model.rb`). When the node has no selective sync configured, the scope is a no-op (all replicables are checksummed), so enabling the flag is safe on instances without selective sync. ## Owners - Most appropriate Slack channel to reach out to: `#g_geo` - Best individual to reach out to: @mkozono ## Expectations ### What are we expecting to happen? Nothing at first, until the instance is configured as a primary Geo site and selective sync is configured. On nodes with selective sync configured, primary checksumming/verification queries are scoped to the selective-sync set, reducing the volume of records verified and the cost of verification metrics queries. On nodes without selective sync, behavior is unchanged (the selective sync scope returns all replicables). ### What can go wrong and how would we detect it? - Checksumming could be incorrectly scoped, causing the source cell to checksum all data. This would be bad on production because there is too much data there. Detect via Geo verification dashboards and the Geo admin verification progress (checksummed/verification totals per replicable). - Verification query performance regressions. Monitor database and Sidekiq dashboards on https://dashboards.gitlab.net (Geo verification workers: `VerificationBatchWorker`, `ReverificationBatchWorker`, `VerificationStateBackfillWorker`). ## Rollout Steps Note: Please make sure to run the chatops commands in the Slack channel that gets impacted by the command. This flag is instance-wide (no actor), so it is enabled globally rather than per actor. ### Rollout on non-production environments - Verify the MR with the feature flag is merged to `master` and has been deployed to non-production environments with `/chatops gitlab run auto_deploy status <merge-commit-of-your-feature>` - [x] Enable the feature globally on non-production environments with `/chatops gitlab run feature set org_mover_extend_selective_sync_to_primary_checksumming true --dev --pre --staging --staging-ref` - [x] Verify that the feature works as expected on a node with selective sync configured. The best environment to validate the feature in is [`staging-canary`](https://about.gitlab.com/handbook/engineering/infrastructure/environments/#staging-canary) as this is the first environment deployed to. Make sure you are [configured to use canary](https://next.gitlab.com/). - [ ] If the feature flag causes end-to-end tests to fail, disable the feature flag on staging to avoid blocking [deployments](https://about.gitlab.com/handbook/engineering/deployments-and-releases/deployments/). ### Specific rollout on production For visibility, all `/chatops` commands that target production must be executed in the [`#production` Slack channel](https://gitlab.slack.com/archives/C101F3796) and cross-posted (with the command results) to the responsible team's Slack channel. - Ensure that the feature MRs have been deployed to both production and canary with `/chatops gitlab run auto_deploy status <merge-commit-of-your-feature>` - [x] This flag has no actor; validate behavior on production by enabling globally (see global rollout) on a controlled schedule, or first confirm behavior on a node with selective sync in non-production. ### Preparation before global rollout - [ ] Set a milestone to this rollout issue to signal for enabling and removing the feature flag when it is stable. - [ ] Check if the feature flag change needs to be accompanied with a [change management issue](https://about.gitlab.com/handbook/engineering/infrastructure-platforms/change-management/#feature-flags-and-the-change-management-process). Cross link the issue here if it does. - [ ] Ensure that you or a representative in development can be available for at least 2 hours after feature flag updates in production. - [ ] Ensure that documentation exists for the feature, and the [version history text](https://docs.gitlab.com/development/documentation/feature_flags/#add-history-text) has been updated. - [ ] Notify the [`#support_gitlab-com` Slack channel](https://gitlab.slack.com/archives/C4XFU81LG) and your team channel. ### Global rollout on production For visibility, all `/chatops` commands that target production must be executed in the [`#production` Slack channel](https://gitlab.slack.com/archives/C101F3796) and cross-posted (with the command results) to the responsible team's Slack channel. - [ ] Enable the feature globally on production with `/chatops gitlab run feature set org_mover_extend_selective_sync_to_primary_checksumming true`. - Monitor the appropriate graphs on https://dashboards.gitlab.net for at least 15 minutes after enabling. - [ ] After the feature has been enabled, wait for [at least one day before releasing the feature](#release-the-feature). ### Release the feature After the feature has been [deemed stable](https://about.gitlab.com/handbook/product-development-flow/feature-flag-lifecycle/#including-a-feature-behind-feature-flag-in-the-final-release), the [clean up](https://docs.gitlab.com/development/feature_flags/controls/#cleaning-up) should be done as soon as possible to permanently enable the feature and reduce complexity in the codebase. - [ ] Create a merge request to remove the `org_mover_extend_selective_sync_to_primary_checksumming` feature flag. The MR should include the following changes: - Remove all references to the feature flag from the codebase. - Remove the YAML definitions for the feature from the repository. - [ ] Ensure that the cleanup MR has been included in the release package. - [ ] Close [the feature issue](https://gitlab.com/gitlab-org/gitlab/-/issues/504138) to indicate the feature will be released in the current milestone. - [ ] Once the cleanup MR has been deployed to production, clean up the feature flag from all environments by running this chatops command in `#production` channel: `/chatops gitlab run feature delete org_mover_extend_selective_sync_to_primary_checksumming --dev --pre --staging --staging-ref --production` - [ ] Close this rollout issue. ## Rollback Steps - [ ] This feature can be disabled on production by running the following Chatops command: ``` /chatops gitlab run feature set org_mover_extend_selective_sync_to_primary_checksumming false ``` - [ ] Disable the feature flag on non-production environments: ``` /chatops gitlab run feature set org_mover_extend_selective_sync_to_primary_checksumming false --dev --pre --staging --staging-ref ``` - [ ] Delete feature flag from all environments: ``` /chatops gitlab run feature delete org_mover_extend_selective_sync_to_primary_checksumming --dev --pre --staging --staging-ref --production ```