## Summary Roll out [the feature](https://gitlab.com/gitlab-org/gitlab/-/work_items/601618) currently behind the `secure_consistency_check_duplicate_contexts` feature flag. - DRI: @ghavenga - Team Slack channel: `#p_non_default_branch_tracking` > [!note] > Process and guidance live in the docs. This issue is just the commands and a place to track the rollout. > "Rolling out" means incrementally enabling the flag on GitLab.com to validate stability. It is not the same as releasing the feature, which happens when the flag is removed. > [Feature flag controls](https://docs.gitlab.com/development/feature_flags/controls/) · [Feature flag lifecycle](https://handbook.gitlab.com/handbook/product-development/how-we-work/product-development-flow/feature-flag-lifecycle/#feature-flag-lifecycle) ## Precondition before you enable the flag Do not enable this flag on any environment until both of these conditions are true: 1. The batched background migration in !259248 has finished on GitLab.com. It marks the extra default tracked contexts as non-default and does not delete data. 1. The DRI decides what this check does after that. The options are to keep it as written, to change it to the same non-destructive behaviour as !259248, or to remove it after the unique index in !256507 is in place. Reason: `DuplicateDefaultTrackedContextsCheck` deletes data. The background migration from !221673 used a similar merge-and-delete approach, and !259248 replaces that migration because the approach is risky and slow. After !259248 finishes, the only duplicates left are the projects that it skips because no branch context matches the default branch. For these projects, this check renames a context and then collapses the duplicates. ## What could go wrong? When the flag is enabled for a project and that project has more than one default tracked context, the check does these steps. The steps do not run in a database transaction. 1. It selects one context to keep. It keeps the context whose name matches the default branch. If no context matches, it keeps the context with the lowest ID and renames it to the default branch. The rename fails if a non-default context with that name already exists (#633935). 1. It deletes the `sbom_occurrence_refs` rows of the other contexts. 1. It moves the `vulnerability_reads` and `vulnerability_occurrences` rows of the other contexts to the kept context. It syncs the moved `vulnerability_reads` rows to Elasticsearch. 1. It moves the `vulnerability_statistics` and `vulnerability_historical_statistics` rows of the other contexts to the kept context. 1. It deletes the other contexts from `security_project_tracked_contexts`. Step 4 can fail. `vulnerability_statistics` has a unique index on `(project_id, security_project_tracked_context_id)`. `vulnerability_historical_statistics` has a unique index on `(project_id, date, security_project_tracked_context_id)`. If the kept context already has a row with the same key, the update fails. Steps 2 and 3 are then complete, and the other contexts are not deleted. The specs do not cover this case. The check does not keep a copy of the deleted rows, or of the original context of the moved rows. The `Collapsed duplicate contexts` log message records only `survivor_id` and `deleted_ids`. To undo a collapse, restore the data from a database backup. Enable the flag for single projects first. For each project, check the log messages `Found duplicate default tracked contexts` and `Collapsed duplicate contexts`. If a project logs the first message but not the second, the collapse failed. ## Events - [Exceptions with secure_consistency_check_duplicate_contexts:1](https://log.gprd.gitlab.net/app/discover#/?_g=(time:(from:now-1d,to:now))&_a=(index:'7092c4e2-4eb5-46f2-8305-a7da2edad090',query:(language:kuery,query:'json.exception.feature_flag_states.keyword:%22secure_consistency_check_duplicate_contexts:1%22'),columns:!(json.exception.class,json.exception.message,json.extra.workflow_id,json.meta.caller_id),sort:!(!(json.time,desc)))) - [Events with secure_consistency_check_duplicate_contexts:1](https://log.gprd.gitlab.net/app/discover#/?_g=(time:(from:now-1d,to:now))&_a=(index:'7092c4e2-4eb5-46f2-8305-a7da2edad090',query:(language:kuery,query:'json.feature_flag_states:%22secure_consistency_check_duplicate_contexts:1%22'),columns:!(json.meta.caller_id,json.feature_flag_states),sort:!(!(json.time,desc)))) - [Error rate and other graphs by modifying the examples in the Visualization Library](https://log.gprd.gitlab.net/app/visualize#/?_g=h@358d019&s=FF%20Observability) Feature Flag events are only logged by default for feature flags marked for the current or future milestones. To enable while the feature flag is active, see https://docs.gitlab.com/development/feature_flags/#logging ## Rollout Run all production `/chatops` in [`#production`](https://gitlab.slack.com/archives/C101F3796) and cross-post the results to `#p_non_default_branch_tracking`. Background: [incremental rollout process](https://docs.gitlab.com/development/feature_flags/controls/#process), [feature actors](https://docs.gitlab.com/development/feature_flags/#feature-actors). **Non-production** ``` /chatops gitlab run feature set secure_consistency_check_duplicate_contexts 50 --actors --dev --pre --staging --staging-ref /chatops gitlab run feature set secure_consistency_check_duplicate_contexts true --dev --pre --staging --staging-ref ``` **Production**: enable for single projects first. The flag uses the project as its actor. ``` /chatops gitlab run feature set --project=<project-path> secure_consistency_check_duplicate_contexts true ``` Then use a percentage rollout (wait at least 15 minutes between steps, and watch the dashboards): ``` /chatops gitlab run feature set secure_consistency_check_duplicate_contexts <percentage> --actors ``` ## Before global rollout Confirm the relevant gotchas before going to 100%. See [enabling a feature for GitLab.com](https://docs.gitlab.com/development/feature_flags/controls/#enabling-a-feature-for-gitlabcom): - [Docs + version history](https://docs.gitlab.com/development/documentation/feature_flags/) updated - [Breaking changes](https://docs.gitlab.com/development/documentation/release_notes/#deprecations-removals-and-breaking-changes) announced, if any - [Change management issue](https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/change-management/#feature-flags-and-the-change-management-process) opened, if required - [External API consumers](https://docs.gitlab.com/development/feature_flags/#do-not-use-feature-flags-in-external-api-consumers) handled with a fail-open mechanism, if applicable ## Cleanup Remove the flag once [deemed stable](https://handbook.gitlab.com/handbook/product-development/how-we-work/product-development-flow/feature-flag-lifecycle/#feature-flag-lifecycle). See [cleaning up](https://docs.gitlab.com/development/feature_flags/controls/#cleaning-up). Track it here, or open a follow-up [Feature Flag Cleanup issue](https://gitlab.com/gitlab-org/gitlab/-/work_items/new?description_template=Feature%20Flag%20Cleanup). Remove the flag and its YAML definition from the codebase, then: ``` /chatops gitlab run release check <merge-request-url> <milestone> /chatops gitlab run feature delete secure_consistency_check_duplicate_contexts --dev --pre --staging --staging-ref --production ``` ## Rollback ``` /chatops gitlab run feature set secure_consistency_check_duplicate_contexts false # production /chatops gitlab run feature set secure_consistency_check_duplicate_contexts false --dev --pre --staging --staging-ref # non-production /chatops gitlab run feature delete secure_consistency_check_duplicate_contexts --dev --pre --staging --staging-ref --production # remove entirely ``` Disabling the flag stops new collapses. It does not restore data that a collapse already deleted or moved.