## Summary Roll out the fix in https://gitlab.com/gitlab-org/gitlab/-/merge_requests/253707 for https://gitlab.com/gitlab-org/gitlab/-/issues/627635 currently behind the `auto_merge_readd_to_train_after_completed_pipeline` feature flag. - DRI: @marc_shaw > [!note] > Process and guidance live in the docs - this issue is just the commands and a place to track the rollout. > "Rolling out" means incrementally enabling the flag on GitLab.com to validate stability - it is not the same as releasing the feature, which happens when the flag is removed. > [Feature flag controls](https://docs.gitlab.com/development/feature_flags/controls/) · [Feature flag lifecycle](https://handbook.gitlab.com/handbook/product-development/how-we-work/product-development-flow/feature-flag-lifecycle/#feature-flag-lifecycle) ## What could go wrong? The flag gates one thing: whether `AutoMerge::AddToMergeTrainWhenChecksPassService#process` skips the CI check when the head pipeline is a finished, unretryable merge train pipeline. - Flag off: today's behaviour exactly. The merge request stays stuck, which is the bug. - Flag on: it goes back on the merge train, which builds a fresh train ref and runs a fresh pipeline. Nothing merges without that new pipeline passing. Blast radius is limited to merge requests that were dropped from a merge train and whose head pipeline is still that train's finished pipeline. Approvals, draft status, discussions, blocking merge requests, external status checks and security policies are all still enforced. Known residual risk: two concurrent `AutoMergeProcessWorker` runs can race. There is a guard immediately before the abort, but it narrows rather than closes the window. If it loses, the symptom is an auto-merge that cancels itself with the system note "removed this merge request from the merge train because the merge request cannot be merged. The pipeline must succeed." Watch for that note during rollout. The concurrency guard is gated on this same flag, so the flag is a full kill switch: with it off, both the CI-check skip and the guard are inert and behaviour is exactly master's. That was verified by running an auto-merge availability matrix (pipeline state x `only_allow_merge_if_pipeline_succeeds` x stored strategy) against the branch and its master parent and diffing the output, and by confirming that removing the guard's flag check breaks the flag-off spec. ## Rollout Run all production `/chatops` in [`#production`](https://gitlab.slack.com/archives/C101F3796). Background: [incremental rollout process](https://docs.gitlab.com/development/feature_flags/controls/#process), [feature actors](https://docs.gitlab.com/development/feature_flags/#feature-actors). **Non-production** ``` /chatops gitlab run feature set auto_merge_readd_to_train_after_completed_pipeline 50 --actors --dev --pre --staging --staging-ref /chatops gitlab run feature set auto_merge_readd_to_train_after_completed_pipeline true --dev --pre --staging --staging-ref ``` **Production** - percentage rollout (wait ≥15 min between steps, watch dashboards): ``` /chatops gitlab run feature set auto_merge_readd_to_train_after_completed_pipeline <percentage> --actors ``` Or target specific actors instead: ``` /chatops gitlab run feature set --project=gitlab-org/gitlab,gitlab-org/gitlab-foss auto_merge_readd_to_train_after_completed_pipeline true /chatops gitlab run feature set --group=gitlab-org,gitlab-com auto_merge_readd_to_train_after_completed_pipeline true /chatops gitlab run feature set --user=marc_shaw auto_merge_readd_to_train_after_completed_pipeline true ``` ## Before global rollout Confirm the relevant gotchas before going to 100% - see [enabling a feature for GitLab.com](https://docs.gitlab.com/development/feature_flags/controls/#enabling-a-feature-for-gitlabcom): - [Docs + version history](https://docs.gitlab.com/development/documentation/feature_flags/) updated - [Breaking changes](https://docs.gitlab.com/development/documentation/release_notes/#deprecations-removals-and-breaking-changes) announced, if any - [Change management issue](https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/change-management/#feature-flags-and-the-change-management-process) opened, if required - [External API consumers](https://docs.gitlab.com/development/feature_flags/#do-not-use-feature-flags-in-external-api-consumers) handled with a fail-open mechanism, if applicable ## Cleanup Remove the flag once [deemed stable](https://handbook.gitlab.com/handbook/product-development/how-we-work/product-development-flow/feature-flag-lifecycle/#feature-flag-lifecycle) - see [cleaning up](https://docs.gitlab.com/development/feature_flags/controls/#cleaning-up). Track it here, or open a follow-up [Feature Flag Cleanup issue](https://gitlab.com/gitlab-org/gitlab/-/work_items/new?description_template=Feature%20Flag%20Cleanup). Remove the flag and its YAML definition from the codebase, then: ``` /chatops gitlab run release check https://gitlab.com/gitlab-org/gitlab/-/merge_requests/253707 19.4 /chatops gitlab run feature delete auto_merge_readd_to_train_after_completed_pipeline --dev --pre --staging --staging-ref --production ``` ## Rollback ``` /chatops gitlab run feature set auto_merge_readd_to_train_after_completed_pipeline false # production /chatops gitlab run feature set auto_merge_readd_to_train_after_completed_pipeline false --dev --pre --staging --staging-ref # non-production /chatops gitlab run feature delete auto_merge_readd_to_train_after_completed_pipeline --dev --pre --staging --staging-ref --production # remove entirely ``` <!-- Uncomment the appropriate type label /label ~"type::feature" ~"feature::addition" /label ~"type::maintenance" /label ~"type::bug" -->