## TL;DR While a security fix is on `security:main` but not yet released, the canonical→security push mirror rejects new canonical commits (`keep_divergent_refs`), and the consumer's deploy pipeline freezes until the fix ships. This issue adds auto-reconciliation to the release platform smart sync: a scheduled pipeline on the security mirror triggers [`gitlab-org/merge-train`](https://gitlab.com/gitlab-org/merge-train) to copy canonical commits into the mirror during that window, and the release-time manual sync converges the branches so the push mirror resumes as the sole writer. This is the pattern gitlab.com uses in [release-tools](https://gitlab.com/gitlab-org/release-tools/-/blob/master/lib/release_tools/security/merge_train_service.rb). Predecessor: [#22049](https://gitlab.com/gitlab-com/gl-infra/delivery/-/work_items/22049). The security→canonical publish path is [#22003](https://gitlab.com/gitlab-com/gl-infra/delivery/-/work_items/22003). ## Design | Piece | Mechanism | |---|---| | Merge execution | `merge-train-trigger` on the security mirror triggers `gitlab-org/merge-train` (`release-platform` mode, [merge-train!84](https://gitlab.com/gitlab-org/merge-train/-/merge_requests/84)) as a child pipeline. `CI_JOB_TOKEN` end to end; no new secrets. | | Trigger gate | `$CI_PIPELINE_SOURCE == "schedule"` plus a `merge_train` CI input. Security mirrors block user-defined CI variables (`no_one_allowed`), so the input, set only by the schedule, is the switch ([why](https://gitlab.com/gitlab-com/gl-infra/common-ci-tasks/-/merge_requests/1754)). | | Schedule | Terraform-managed on the mirror: hourly, inactive by default, `MERGE_TRAIN_SCHEDULE_ID` project variable. Per-consumer `merge_train_enabled` flag in infra-mgmt `repos.yaml`. | | Identity | Only the release bot may push `security:main`, and schedule pipelines run as the schedule owner. Activation therefore runs `take_ownership` as the bot; this also self-heals accidental human ownership. | | Activation | `generate-publish-pipeline`, when the release commit is only on the security mirror (`SYNC_TYPE=manual`): take ownership, set `active=true`, and ensure `crossProjectPushForJobTokenAllowed` on the mirror (GraphQL-only setting, GitLab 19.0). | | Release of the fix | The operator plays the manual sync jobs. `sync-commits` refuses security commits that have not been released (scope guard), plays one merge-train pipeline as the bot when canonical moved, then fast-forwards canonical to the security head. | | Deactivation | `merge-train-deactivate` (needs both sync jobs) sets `active=false` once canonical contains the security head. | Deactivation at release time is a correctness requirement: after convergence the push mirror resumes, and an active schedule would be a competing writer that can re-diverge the mirror. ## Status (2026-08-24) Validated end to end twice on the canary: - Cycle 1 (`v1.7.2`, tag pipeline created manually): [E2E evidence comment](https://gitlab.com/gitlab-com/gl-infra/delivery/-/work_items/22136#note_3679610739) - Cycle 2 (`v1.7.4`, zero manual workarounds): walkthrough below. All findings are fixed, merged, and released: | MR | Fixes | Released | |---|---|---| | [common-ci-tasks!1776](https://gitlab.com/gitlab-com/gl-infra/common-ci-tasks/-/merge_requests/1776) | F1: `sync-commits` manual sync redesign. Auto releases push exactly the release SHA; manual releases refuse security commits that have not been released, converge through a merge-train run, and fast-forward canonical. | `v4.45.0` | | [common-ci-tasks!1777](https://gitlab.com/gitlab-com/gl-infra/common-ci-tasks/-/merge_requests/1777) | F2: schedule activation ensures `crossProjectPushForJobTokenAllowed` via GraphQL. The setting has no REST parameter and no Terraform provider attribute (verified through provider v19.2.1). | `v4.46.0` | | [common-ci-tasks!1789](https://gitlab.com/gitlab-com/gl-infra/common-ci-tasks/-/merge_requests/1789) | semantic-release pushes with the configured token, never the job token. Job-token pushes create no pipelines; this caused the `v1.7.3` tag-without-pipeline release ([#22280](https://gitlab.com/gitlab-com/gl-infra/delivery/-/work_items/22280)). | `v5.1.2` | ### Validation cycle 2: clean end-to-end (2026-08-24, `v1.7.4`) One `fix:` MR on the mirror drove the entire lifecycle with no manual workaround: 1. Release MR on the mirror: [security!23](https://gitlab.com/gitlab-org/security/release-platform-canary/-/merge_requests/23), merged via MWPS. 2. semantic-release on `main` ([pipeline 2786756367](https://gitlab.com/gitlab-org/security/release-platform-canary/-/pipelines/2786756367)) tagged `v1.7.4` as `project_79940647_bot_*`, through the configured token. 3. The tag got a pipeline ([2786757660](https://gitlab.com/gitlab-org/security/release-platform-canary/-/pipelines/2786757660)) -- the thing `v1.7.3` never got. In it, [`generate-publish-pipeline`](https://gitlab.com/gitlab-org/security/release-platform-canary/-/jobs/16076889605) activated schedule `4385554` as the bot and ensured `crossProjectPushForJobTokenAllowed`. 4. goreleaser filled the release: [v1.7.4 on the mirror](https://gitlab.com/gitlab-org/security/release-platform-canary/-/releases/v1.7.4), 21 assets. 5. Canonical moved while the fix was unreleased ([canary!216](https://gitlab.com/gitlab-org/software-delivery/release-platform-canary/-/merge_requests/216)); the hourly tick ([pipeline 2786955531](https://gitlab.com/gitlab-org/security/release-platform-canary/-/pipelines/2786955531)) copied it to the mirror. 6. Release time: the manual sync jobs on the [publish child 2786766508](https://gitlab.com/gitlab-org/security/release-platform-canary/-/pipelines/2786766508) synced artifacts, commits, the default branch, and the release page. Canonical fast-forwarded to the security head: [v1.7.4 on canonical](https://gitlab.com/gitlab-org/software-delivery/release-platform-canary/-/releases/v1.7.4). 7. `merge-train-deactivate` set the schedule inactive (the fix is public) and the push mirror resumed on its own. This also closes the main acceptance criterion of [#22280](https://gitlab.com/gitlab-com/gl-infra/delivery/-/work_items/22280): a bot-identity tag with a tag pipeline. Remaining: 1. ~~Renovate bumps the canary~~ Done: [canary!209](https://gitlab.com/gitlab-org/software-delivery/release-platform-canary/-/merge_requests/209) merged 2026-08-20; the canary is on `v5.1` (F1, F2, and the [semantic-release token fix](https://gitlab.com/gitlab-com/gl-infra/common-ci-tasks/-/merge_requests/1789)). A 2026-08-18 merge attempt reported success but did not merge (missing approval); releases `v1.7.3` and earlier ran on `v4.40.0`. The tube v5 migration for the remaining consumers is tracked in [delivery#22310](https://gitlab.com/gitlab-com/gl-infra/delivery/-/work_items/22310). 2. ~~Run one canary security-fix release cycle end to end with zero manual steps~~ Done 2026-08-24 on `v1.7.4`; walkthrough above. 3. ~~Restore the approval settings on the canary projects~~ Done 2026-08-24 (the mirror PSIRT rule count still needs to go back to 1; the release-platform module manages it, so the next infra-mgmt apply also restores it). 4. ~~Rollout: flip `merge_train_enabled` per consumer in `repos.yaml`~~ Done 2026-08-25: [infra-mgmt!3184](https://gitlab.com/gitlab-com/gl-infra/infra-mgmt/-/merge_requests/3184) merged and applied (56 added / 18 changed / 29 destroyed, matching the plan). Verified on all 9 mirrors: inactive `release-framework merge-train` schedule + matching `MERGE_TRAIN_SCHEDULE_ID` variable. The canary PSIRT approval rule is back at 1 (module-managed). All deliverables are done; the issue is closed. Notes for consumer teams (informational, not deliverables): - The rollout apply replaced all 10 canonical→security push mirrors (URL drift from the August token rotation). A replaced mirror resyncs on its canonical's next push. - Job-token push is now enabled on the mirrors. Consumers pinned below common-ci-tasks `v5.1.2` (gate-charts `v3.10`, membrane `v4.43`, sts `v4.19`) hit the silent hollow-release bug ([#22280](https://gitlab.com/gitlab-com/gl-infra/delivery/-/work_items/22280)) if they release from the security mirror before their Renovate bump. Bumping is each team's call, on their own schedule. - Consumer schedules are owned by `gitlab-infra-mgmt-bot` until first activation runs `take_ownership` as each release bot (same lifecycle as the canary; expected). - knowledge-graph uses a custom release path (no semantic-release component); the note above does not apply to it. ## Merged work and history <details><summary>Merged MRs, design pivot, POC, incident</summary> | Change | MR | State | |---|---|---| | Terraform substrate: job-token allowlists, `merge_train_enabled` in `repos.yaml` | [infra-mgmt!2898](https://gitlab.com/gitlab-com/gl-infra/infra-mgmt/-/merge_requests/2898) | merged + applied | | merge-train `release-platform` mode | [merge-train!84](https://gitlab.com/gitlab-org/merge-train/-/merge_requests/84) | merged | | Component trigger job | [common-ci-tasks!1672](https://gitlab.com/gitlab-com/gl-infra/common-ci-tasks/-/merge_requests/1672) | merged (`v4.30.0`; canary bump [canary!174](https://gitlab.com/gitlab-org/software-delivery/release-platform-canary/-/merge_requests/174)) | | Gate on a CI input instead of a variable | [common-ci-tasks!1754](https://gitlab.com/gitlab-com/gl-infra/common-ci-tasks/-/merge_requests/1754) | merged | | Schedule + bot Maintainer on the mirror | [release-platform!102](https://gitlab.com/gitlab-com/gl-infra/terraform-modules/gitlab/release-platform/-/merge_requests/102) | merged (module `v5.6.0`) | | Consumer input forwarding | [canary!197](https://gitlab.com/gitlab-org/software-delivery/release-platform-canary/-/merge_requests/197) | merged (`v4.40.0`) | | infra-mgmt module bump | [infra-mgmt!3132](https://gitlab.com/gitlab-com/gl-infra/infra-mgmt/-/merge_requests/3132) | merged + applied (`v5.6.1`, includes the schedule ref fix [release-platform!104](https://gitlab.com/gitlab-com/gl-infra/terraform-modules/gitlab/release-platform/-/merge_requests/104)) | | Schedule lifecycle (activate + deactivate) | [common-ci-tasks!1756](https://gitlab.com/gitlab-com/gl-infra/common-ci-tasks/-/merge_requests/1756) | merged (`v4.40.0`) | | E2E fix: bot group role | [release-platform!105](https://gitlab.com/gitlab-com/gl-infra/terraform-modules/gitlab/release-platform/-/merge_requests/105) | merged + applied (`v5.6.2` via [infra-mgmt!3133](https://gitlab.com/gitlab-com/gl-infra/infra-mgmt/-/merge_requests/3133)) | | E2E fix: pipeline creation permission on merge-train `master` | [infra-mgmt!3134](https://gitlab.com/gitlab-com/gl-infra/infra-mgmt/-/merge_requests/3134) | merged + applied | | E2E restore after the stale pipeline-retry revert | [infra-mgmt!3137](https://gitlab.com/gitlab-com/gl-infra/infra-mgmt/-/merge_requests/3137) | merged + applied (schedule recreated as `4385554`) | - Design pivot (2026-06): reuse `gitlab-org/merge-train` as a child pipeline instead of reimplementing merge logic in the component ([suggestion on !1594](https://gitlab.com/gitlab-com/gl-infra/common-ci-tasks/-/merge_requests/1594#note_3451724575); !1594 closed, branch preserved). - POC: `CI_JOB_TOKEN` identity inherits across the cross-project trigger boundary, so no new service account is needed. Results and failure-mode table: [POC comment](https://gitlab.com/gitlab-com/gl-infra/delivery/-/work_items/22136#note_3457892861). - Operational caveats: playing a schedule runs it as the API caller, not the owner (the manual sync relies on this: it plays the schedule with the bot token); a recreated push mirror does not sync until the next push or a manual force-sync. - Incident (2026-08-12): a retry of a stale failed infra-mgmt `main` pipeline re-applied all environments at a 5-day-old commit and reverted the canary substrate. Restored by [infra-mgmt!3137](https://gitlab.com/gitlab-com/gl-infra/infra-mgmt/-/merge_requests/3137). - Mirror tag pipelines stopped after `v1.7.1` (semantic-release PRAT regression, [#22280](https://gitlab.com/gitlab-com/gl-infra/delivery/-/work_items/22280)); the `v1.7.2` E2E pipeline was created manually. </details> ## Follow-ups (not blocking) - [#22271](https://gitlab.com/gitlab-com/gl-infra/delivery/-/work_items/22271) owned CI image, [#22267](https://gitlab.com/gitlab-com/gl-infra/delivery/-/work_items/22267) submodule extraction, [#22280](https://gitlab.com/gitlab-com/gl-infra/delivery/-/work_items/22280) semantic-release PRAT regression. - Pre-existing `sync-commits`/`sync-default-branch` hardening list, documented in the !1776 review threads. - Upstream feature gap: `crossProjectPushForJobTokenAllowed` has no REST parameter or Terraform provider attribute. When the provider gains it, the module manages the setting and the component's GraphQL ensure becomes a no-op safety net. ## Out of scope - Replacing the manual sync gate: canonical is public, so the operator decides when a security commit goes public. - Un-tagged divergence (a security commit that cuts no tag never activates the schedule): rare; the operator can activate the schedule via UI or API. - Migrating gitlab.com itself. ## Resources - gitlab.com implementation: [merge_train_service.rb](https://gitlab.com/gitlab-org/release-tools/-/blob/master/lib/release_tools/security/merge_train_service.rb), [sync_remotes_service.rb](https://gitlab.com/gitlab-org/release-tools/-/blob/master/lib/release_tools/security/sync_remotes_service.rb) - Docs: [mirrors.md](https://gitlab.com/gitlab-org/release/docs/-/blob/master/general/security/mirrors.md), [merge_train.md](https://gitlab.com/gitlab-org/release/docs/-/blob/master/general/security/utilities/merge_train.md), [sync-failure runbook](https://gitlab.com/gitlab-org/release/docs/-/blob/master/runbooks/security/resolving-sync-failures.md) ## Acceptance criteria - [x] While a security fix is unreleased, canonical commits keep reaching `security:main` through the scheduled merge-train; the deploy pipeline does not freeze. - [x] The schedule auto-activates when the publish flow detects that the release commit is only on the security mirror (`SYNC_TYPE=manual`). Validated live on the `v1.7.2` release. - [x] The schedule auto-deactivates when the fix reaches canonical, so the push mirror resumes as the sole writer. Validated live; hands-off convergence shipped in [!1776](https://gitlab.com/gitlab-com/gl-infra/common-ci-tasks/-/merge_requests/1776) (`v4.45.0`). - [x] Provisioning is automated: Terraform for everything the provider supports, and the component ensures the GraphQL-only push setting ([!1777](https://gitlab.com/gitlab-com/gl-infra/common-ci-tasks/-/merge_requests/1777), `v4.46.0`). - [x] Validated end to end on `release-platform-canary`.