[FF] auto_merge_readd_to_train_after_completed_pipeline - stop auto-merge hanging on a finished merge train pipeline

Summary

Roll out the fix in !253707 (merged) for #627635 (closed) currently behind the auto_merge_readd_to_train_after_completed_pipeline feature flag.

Note

Process and guidance live in the docs - this issue is just the commands and a place to track the rollout. "Rolling out" means incrementally enabling the flag on GitLab.com to validate stability - it is not the same as releasing the feature, which happens when the flag is removed. Feature flag controls · Feature flag lifecycle

What could go wrong?

The flag gates one thing: whether AutoMerge::AddToMergeTrainWhenChecksPassService#process skips the CI check when the head pipeline is a finished, unretryable merge train pipeline.

  • Flag off: today's behaviour exactly. The merge request stays stuck, which is the bug.
  • Flag on: it goes back on the merge train, which builds a fresh train ref and runs a fresh pipeline. Nothing merges without that new pipeline passing.

Blast radius is limited to merge requests that were dropped from a merge train and whose head pipeline is still that train's finished pipeline. Approvals, draft status, discussions, blocking merge requests, external status checks and security policies are all still enforced.

Known residual risk: two concurrent AutoMergeProcessWorker runs can race. There is a guard immediately before the abort, but it narrows rather than closes the window. If it loses, the symptom is an auto-merge that cancels itself with the system note "removed this merge request from the merge train because the merge request cannot be merged. The pipeline must succeed." Watch for that note during rollout.

The concurrency guard is gated on this same flag, so the flag is a full kill switch: with it off, both the CI-check skip and the guard are inert and behaviour is exactly master's. That was verified by running an auto-merge availability matrix (pipeline state x only_allow_merge_if_pipeline_succeeds x stored strategy) against the branch and its master parent and diffing the output, and by confirming that removing the guard's flag check breaks the flag-off spec.

Rollout

Run all production /chatops in #production. Background: incremental rollout process, feature actors.

Non-production

/chatops gitlab run feature set auto_merge_readd_to_train_after_completed_pipeline 50 --actors --dev --pre --staging --staging-ref
/chatops gitlab run feature set auto_merge_readd_to_train_after_completed_pipeline true --dev --pre --staging --staging-ref

Production - percentage rollout (wait ≥15 min between steps, watch dashboards):

/chatops gitlab run feature set auto_merge_readd_to_train_after_completed_pipeline <percentage> --actors

Or target specific actors instead:

/chatops gitlab run feature set --project=gitlab-org/gitlab,gitlab-org/gitlab-foss auto_merge_readd_to_train_after_completed_pipeline true
/chatops gitlab run feature set --group=gitlab-org,gitlab-com auto_merge_readd_to_train_after_completed_pipeline true
/chatops gitlab run feature set --user=marc_shaw auto_merge_readd_to_train_after_completed_pipeline true

Before global rollout

Confirm the relevant gotchas before going to 100% - see enabling a feature for GitLab.com:

Cleanup

Remove the flag once deemed stable - see cleaning up. Track it here, or open a follow-up Feature Flag Cleanup issue. Remove the flag and its YAML definition from the codebase, then:

/chatops gitlab run release check https://gitlab.com/gitlab-org/gitlab/-/merge_requests/253707 19.4
/chatops gitlab run feature delete auto_merge_readd_to_train_after_completed_pipeline --dev --pre --staging --staging-ref --production

Rollback

/chatops gitlab run feature set auto_merge_readd_to_train_after_completed_pipeline false                                         # production
/chatops gitlab run feature set auto_merge_readd_to_train_after_completed_pipeline false --dev --pre --staging --staging-ref     # non-production
/chatops gitlab run feature delete auto_merge_readd_to_train_after_completed_pipeline --dev --pre --staging --staging-ref --production  # remove entirely
Edited by Marc Shaw