## TL;DR Add a `sync-commits` job to the publish-stage child pipeline that fast-forward-pushes security HEAD + the release tag to canonical's `main` after artifacts sync, so canonical's git history converges with security's. Auth via the existing per-project `release-bot` group SA (scopes expanded to include `read_repository` + `write_repository`; user added to canonical's branch and tag protection allow lists). ## Problem / Goal The publish-stage smart sync ([delivery#22003](https://gitlab.com/gitlab-com/gl-infra/delivery/-/work_items/22003)) syncs **artifacts** (container images, OCI Helm charts, generic packages) from the security mirror to canonical when a tagged commit is approved for publication. But the **git commits** that produced those artifacts stay on the security mirror only. After every security release, canonical's git history falls behind security's by one (or more) commits, and the only way to align them is manual force-push or cherry-pick. This issue tracks automating the source-side reconciliation so the two repos re-converge after artifacts land on canonical. ## Background / Context The same problem class showed up in canary testing for delivery#22003: a fix merged into security's `main` without a corresponding commit on canonical's `main` causes the `canonical→security` push mirror to permanently skip `main` (because of `keep_divergent_refs = true`). Real security fixes have the same shape; the canary just made it visible. Without a sync-commits step, every security release leaves canonical's `main` stale until someone manually reconciles. **Current state (post-delivery#22003):** the publish-stage smart sync ships in common-ci-tasks `v4.0.0` with a single `trigger-publish-pipeline` job (`strategy: depend`) and a child pipeline containing `sync-start` (no-op auto-job that lifts the child out of `skipped`), `sync-registry-artifacts`, and `sync-packages` (both gated by `when: ${SYNC_WHEN}` -- `on_success` for the auto path, `manual` for the security-only path). The sync-commits work in this issue extends that child pipeline. ## Proposed Solution Add a `sync-commits` job to the publish stage's child pipeline in common-ci-tasks, on its own stage that runs **after** the artifact sync stage. The maintainer plays artifacts first, then commits -- natural binaries-then-source order. **Child pipeline restructure:** - Rename current `sync` stage to `[sync-artifacts, sync-commits]` - `sync-start` and `sync-registry-artifacts` and `sync-packages` move to `stage: sync-artifacts` - New `sync-commits` job in `stage: sync-commits` - `sync-commits` is **conditionally emitted only when `SYNC_TYPE == "manual"`**. The auto path means the commit + tag already exist on canonical (semantic-release pushed to canonical first, then mirrored canonical→security), so commit sync is a no-op. Manual path is the only case where canonical is missing the commit + tag. - The `sync-start` no-op auto-job continues to lift the child pipeline out of `skipped` for the manual case; sync-commits inherits that **Auth: extend the existing per-project `release-bot` group SA.** The release-platform Terraform module already provisions a per-project group service account `<path>-release-bot` that drives the canonical→security push mirror. The same SA also authorises sync-commits: - Existing SA stays Developer-scoped on canonical and Developer-shared into security - The SA's access token's `scopes` expand from `["api"]` to `["api", "read_repository", "write_repository"]` so the same token can authorize git push to canonical - The SA's `service_account_id` is added to canonical's `gitlab_branch_protection.allowed_to_push` for `main` and `gitlab_tag_protection.allowed_to_create` for `v*` tags so the SA can bypass `push_access_level = "no one"` and `tag_create_access_level = "maintainer"` - The SA's username + token are exposed on the security mirror as masked + protected CI/CD variables `CANONICAL_REPO_TOKEN_USERNAME` and `CANONICAL_REPO_TOKEN` (consumed by the `sync-commits` job) - The job uses HTTPS-with-token to `git push` the commits and tag **Push strategy is fast-forward only:** - Job fetches canonical's `main`, checks that canonical's HEAD is an ancestor of the security HEAD, then pushes the security HEAD as canonical's `main` and pushes the tag - If they've diverged (canonical has commits security doesn't), the job fails with a clear message and the maintainer reconciles manually - The automation never overwrites canonical history ## Out of Scope - Changing the auto-vs-manual decision logic from delivery#22003 (continues to be: HTTP 200 on the commit-on-canonical check → auto, anything else → manual) - Orchestrating multi-tag backfill (only handles the most recent tag's commits in fast-forward order) - Resolving divergence automatically (cherry-pick / rebase / merge — those remain manual) ## Affected Systems - [`gitlab-com/gl-infra/common-ci-tasks`](https://gitlab.com/gitlab-com/gl-infra/common-ci-tasks) -- `src/ci/components/release-platform.jsonnet` (heredoc that emits `sync-pipeline.yml`) - [`gitlab-com/gl-infra/terraform-modules/gitlab/release-platform`](https://gitlab.com/gitlab-com/gl-infra/terraform-modules/gitlab/release-platform) -- expand the existing `release-bot` SAAT scopes, add the SA to canonical's `main` branch protection and `v*` tag protection allow lists, expose the SA's username + token on the security mirror as `CANONICAL_REPO_TOKEN{,_USERNAME}` masked CI/CD variables - [`gitlab-com/gl-infra/infra-mgmt`](https://gitlab.com/gitlab-com/gl-infra/infra-mgmt) -- consumes the new release-platform module version (Renovate-managed) - [`gitlab-org/software-delivery/release-platform-canary`](https://gitlab.com/gitlab-org/software-delivery/release-platform-canary) and its security mirror -- validation target ## Resources - Parent issue: [delivery#22003](https://gitlab.com/gitlab-com/gl-infra/delivery/-/work_items/22003) - Parent epic: [software-delivery#51](https://gitlab.com/groups/gitlab-com/gl-infra/software-delivery/-/work_items/51) - Existing pattern (registry-sync deploy token): [release-platform!41](https://gitlab.com/gitlab-com/gl-infra/terraform-modules/gitlab/release-platform/-/merge_requests/41) and [release-platform!44](https://gitlab.com/gitlab-com/gl-infra/terraform-modules/gitlab/release-platform/-/merge_requests/44) - Existing pattern (publish stage smart sync): [common-ci-tasks!1437](https://gitlab.com/gitlab-com/gl-infra/common-ci-tasks/-/merge_requests/1437) and [common-ci-tasks!1443](https://gitlab.com/gitlab-com/gl-infra/common-ci-tasks/-/merge_requests/1443) - Group SA pattern (custom_mirror_token, v3.0.0 release-platform): [release-platform!43](https://gitlab.com/gitlab-com/gl-infra/terraform-modules/gitlab/release-platform/-/merge_requests/43) ## Tasks - [x] Emission design: **conditional emit only when `SYNC_TYPE == "manual"`** (auto path has commit + tag on canonical already → nothing to sync) - [x] Expand the existing `gitlab_group_service_account_access_token.bot.scopes` from `["api"]` to `["api", "read_repository", "write_repository"]` so the same token can authorize git push to canonical -- in [release-platform!50](https://gitlab.com/gitlab-com/gl-infra/terraform-modules/gitlab/release-platform/-/merge_requests/50) - [x] Add the existing `release-bot` SA's `service_account_id` to canonical's `gitlab_branch_protection.allowed_to_push` for `main` and `gitlab_tag_protection.allowed_to_create` for `v*` tags -- in [release-platform!50](https://gitlab.com/gitlab-com/gl-infra/terraform-modules/gitlab/release-platform/-/merge_requests/50) - [x] Expose the existing `release-bot` SA's username + token on the security mirror as `CANONICAL_REPO_TOKEN_USERNAME` and `CANONICAL_REPO_TOKEN` (both masked + protected) -- in [release-platform!50](https://gitlab.com/gitlab-com/gl-infra/terraform-modules/gitlab/release-platform/-/merge_requests/50) - [x] Tag a new release-platform module version (after !50 merges) -- shipped as `v4.1.0` (after a no-op `feat:` follow-up in [release-platform!53](https://gitlab.com/gitlab-com/gl-infra/terraform-modules/gitlab/release-platform/-/merge_requests/53) since !50 was a `refactor:` and didn't trigger semantic-release on its own) - [x] Bump infra-mgmt to `v4.1.1` and apply -- via [infra-mgmt!2766](https://gitlab.com/gitlab-com/gl-infra/infra-mgmt/-/merge_requests/2766) (required state surgery to reconcile GitLab-side SA purges + token rotations; v4.1.0 had a bug fixed in [release-platform!55](https://gitlab.com/gitlab-com/gl-infra/terraform-modules/gitlab/release-platform/-/merge_requests/55) → v4.1.1) - [x] Restructure the publish stage's child pipeline stages from `[sync]` to `[sync-artifacts, sync-commits]`; move `sync-start`, `sync-registry-artifacts`, `sync-packages` to `stage: sync-artifacts` -- in [common-ci-tasks!1498](https://gitlab.com/gitlab-com/gl-infra/common-ci-tasks/-/merge_requests/1498) - [x] Add the `sync-commits` job in `stage: sync-commits` with fast-forward-only push logic (fetch canonical's main, ancestor check, push commit + tag) -- in [common-ci-tasks!1498](https://gitlab.com/gitlab-com/gl-infra/common-ci-tasks/-/merge_requests/1498) - [x] Reuse `canonical_registry_path` as the project path (registry path == project path on GitLab; no new input needed) - [x] Update `release-platform.md` to document the new child-pipeline structure and the `CANONICAL_REPO_TOKEN_*` variables -- in [common-ci-tasks!1498](https://gitlab.com/gitlab-com/gl-infra/common-ci-tasks/-/merge_requests/1498). Terraform module's `README.md` already documented the variables in [release-platform!50](https://gitlab.com/gitlab-com/gl-infra/terraform-modules/gitlab/release-platform/-/merge_requests/50) - [x] Fix curl API auth: switch sync-commits + sync-default-branch HTTP API calls from `CI_JOB_TOKEN` to `CANONICAL_REPO_TOKEN`; add `api_get` diagnostic helper that surfaces URL + HTTP status + response body on failure -- in [common-ci-tasks!1541](https://gitlab.com/gitlab-com/gl-infra/common-ci-tasks/-/merge_requests/1541) - [x] Fix SYNC_TYPE detection + git push auth: switch `generate-publish-pipeline` checks to `CANONICAL_REPO_TOKEN`; switch git push from PRIVATE-TOKEN header to HTTP Basic auth (Group Service Account Access Tokens require it) -- in [common-ci-tasks!1545](https://gitlab.com/gitlab-com/gl-infra/common-ci-tasks/-/merge_requests/1545) - [x] Fix canonical-path always-manual regression: semantic-release tags only on security mirror, so the tag check on canonical can never return 200; SYNC_TYPE now keys on commit visibility alone -- in [common-ci-tasks!1550](https://gitlab.com/gitlab-com/gl-infra/common-ci-tasks/-/merge_requests/1550) - [x] Validate end-to-end (security path) on `release-platform-canary`: security-only commit -> tag -> maintainer plays sync jobs -> canonical's main converges and canonical has the tag. Confirmed via [pipeline 2567788896](https://gitlab.com/gitlab-org/security/release-platform-canary/-/pipelines/2567788896) for v1.5.1. - [x] Validate auto path (canonical path) on `release-platform-canary`: commit on canonical, tag not yet on canonical -> SYNC_TYPE=auto, all sync jobs run with `when: on_success` and push the tag + align main automatically. Confirmed via [pipeline 2568099522](https://gitlab.com/gitlab-org/security/release-platform-canary/-/pipelines/2568099522) for v1.5.2. - [x] Validate divergent-state failure mode (canonical has a commit security doesn't -> `sync-commits` refuses with fast-forward error and falls back to MR) - [x] Address `sync-start` false-green UX: child pipeline reports `success` after the auto-startable `sync-start` no-op even when manual sync jobs haven't been played. `sync-start` itself is load-bearing (without it, manual-path child gets marked `skipped` and propagates as parent failure under `strategy: depend` -- see [reply 3407398034](https://gitlab.com/gitlab-com/gl-infra/delivery/-/work_items/22049#note_3407398034)). Mitigation candidates: rename `sync-start` (e.g. `sync-gate`) to reduce success-implying connotation; emit a prominent "MANUAL SYNC PENDING" banner in `sync-start`'s log; or surface manual job presence in the parent pipeline view. Per [skarbek note 3407377396](https://gitlab.com/gitlab-com/gl-infra/delivery/-/work_items/22049#note_3407377396). ## Status / Progress Mostly complete -- divergent-state validation remains. | Workstream | Status | |---|---| | Predecessor gitlab-com/gl-infra/delivery#22003 (publish-stage smart sync) | ✅ closed; ships in common-ci-tasks `v4.0.0` | | Build-mirror removal applied across all 6 release-platform consumers via gitlab-com/gl-infra/infra-mgmt!2763 | ✅ applied + merged | | Terraform module sync-commits SA work via [release-platform!50](https://gitlab.com/gitlab-com/gl-infra/terraform-modules/gitlab/release-platform/-/merge_requests/50) | ✅ merged → `v4.1.0` (after no-op `feat:` follow-up in [release-platform!53](https://gitlab.com/gitlab-com/gl-infra/terraform-modules/gitlab/release-platform/-/merge_requests/53)) | | `bot_canonical` share fix via [release-platform!55](https://gitlab.com/gitlab-com/gl-infra/terraform-modules/gitlab/release-platform/-/merge_requests/55) (canonical project-membership required for SA allow-listing) | ✅ merged → `v4.1.1` | | infra-mgmt bump to `release-platform v4.1.1` via gitlab-com/gl-infra/infra-mgmt!2766 | ✅ applied + merged (required state surgery to reconcile GitLab-side SA purges + token rotations + stale tag protections) | | Live state per consumer: SA scopes expanded (`api`+`read_repository`+`write_repository`), SA in canonical `main`/`v*` allow lists, `CANONICAL_REPO_TOKEN_*` variables on security mirror, canonical→security push mirror restored | ✅ verified | | common-ci-tasks `sync-commits` job (consumer-side CI work) via [common-ci-tasks!1498](https://gitlab.com/gitlab-com/gl-infra/common-ci-tasks/-/merge_requests/1498) | ✅ merged → `v4.4.0` | | common-ci-tasks!1541 (CANONICAL_REPO_TOKEN for API calls + `api_get` diagnostic helper) | ✅ merged | | common-ci-tasks!1545 (SYNC_TYPE detection + git push SAAT Basic auth) | ✅ merged | | common-ci-tasks!1550 (auto-sync when commit exists on canonical regardless of tag) | ✅ merged → `v4.5.3` | | End-to-end: security path (commit only on security) | ✅ via [pipeline 2567788896](https://gitlab.com/gitlab-org/security/release-platform-canary/-/pipelines/2567788896) (v1.5.1) and re-validated via [pipeline 2568133347](https://gitlab.com/gitlab-org/security/release-platform-canary/-/pipelines/2568133347) (v1.5.3, post-!1550) -- sync jobs run as manual; operator plays them; canonical:main converges, tag pushed | | End-to-end: canonical path (commit on canonical, tag missing on canonical) | ✅ via [pipeline 2568099522](https://gitlab.com/gitlab-org/security/release-platform-canary/-/pipelines/2568099522) (v1.5.2) -- all 5 sync jobs ran automatically (`when: on_success`); canonical:main + tag converged without operator intervention | | Divergent-state fail-safe (canonical has a commit security doesn't) | ✅ | | `sync-start` false-green UX (job is load-bearing; UX of the success banner is the open question) | ⬜ open follow-up per [skarbek note 3407377396](https://gitlab.com/gitlab-com/gl-infra/delivery/-/work_items/22049#note_3407377396) and [reply 3407398034](https://gitlab.com/gitlab-com/gl-infra/delivery/-/work_items/22049#note_3407398034) | ## Acceptance Criteria - [x] When a tagged commit on the security mirror is **only on the security mirror**, the publish stage's child pipeline includes a `when: manual` `sync-commits` job in a `sync-commits` stage that runs after `sync-artifacts` - [x] When a tagged commit is **already on canonical** (SYNC_TYPE=auto), `sync-commits` and `sync-default-branch` run automatically (`when: on_success`) and push the tag + align main without operator intervention (`sync-commits` is always emitted; `when:` resolves from `SYNC_TYPE` per [common-ci-tasks!1550](https://gitlab.com/gitlab-com/gl-infra/common-ci-tasks/-/merge_requests/1550)) - [x] Playing `sync-commits` performs a fast-forward push of the security HEAD to canonical's `main` AND pushes the git tag, when the histories are linear - [x] If canonical has diverged from security, `sync-commits` fails with a clear message and does not overwrite canonical history - [x] The existing `release-bot` SA's expanded scopes, branch + tag protection bypass, and the security mirror's `CANONICAL_REPO_TOKEN_USERNAME` / `CANONICAL_REPO_TOKEN` variables are all provisioned automatically by the `release-platform` Terraform module -- no per-project manual setup - [ ] Validated end-to-end on `release-platform-canary` for both the linear (success) and divergent (fail-safe) cases