## Problem `common-ci-tasks` consumers are generally hosted on one instance and mirrored to another (gitlab.com → ops.gitlab.net for infra projects; canonical → `security/` mirror and dev.gitlab.org for the Release Platform). Today every validation job in our templates (`go_unittests`, `golangci_lint`, `shellcheck`, `rubocop`, `terraform_validate`, …) runs on **every** instance the pipeline lands on. This has two costs: 1. **Money.** The same commit is linted and tested twice (or three times) for identical, deterministic results. 2. **Fragility.** Anything a validation job pulls in that only exists on the canonical instance breaks the mirror. gitlab-com/gl-infra/tenant-scale/cells-infrastructure/team#846 (confidential) is the worked example: `golang-build@v4.35` started including `modular-feature-testing/go-test` unconditionally. That component exists only on gitlab.com, so on ops.gitlab.net config assembly failed with `content not found` and a **zero-job pipeline**. terra-transformer was red on ops for ~3 weeks, goreleaser never published, and the `dedicated-container-image` → Instrumentor → Cells/Tissue chain starved from an empty ops registry. Nobody noticed because gitlab.com was green. The fix that landed (!1785) is correct but bespoke: one `rules: [{ if: $CI_SERVER_HOST == "gitlab.com" }]` on one include for one task. Meanwhile three consumers already carry their own workarounds: gitlab-com/gl-infra/terra-transformer!906 inlines `golang-build`'s sub-templates minus `go-test` and must now track that include list by hand; gitlab-com/gl-infra/reaper!11 and gitlab-com/gl-infra/platform/runway/runwayctl!2002 hit the same thing. Each new canonical-only dependency will repeat this cycle unless the rule is systematic. ## Proposal > **All deterministic validation jobs in `common-ci-tasks` are skipped on a mirror unless the consumer explicitly opts in.** Where: - A **deterministic validation job** is one whose outcome depends only on the repository contents at `$CI_COMMIT_SHA` and the pinned toolchain: linters, formatters, unit tests, schema/policy checks, `go mod tidy` drift, `terraform validate`, etc. It has no side effects, publishes nothing, and produces the same result on any instance. - A **mirror** is a project that receives its refs from a canonical project on another instance (or namespace) where the same pipeline has already run. - The **opt-in flag** re-enables validation on the mirror for the projects that need it (see *Why an opt-in and not a hard rule*). This is the same shape as `release_platform_enabled` (instance-aware `when: never` guards) but applied once, centrally, to the shared rule set instead of per-job. ## Design ### 1. One predicate, in `src/ci/lib/rules.libsonnet` Add a mirror guard alongside `notScheduled` / `notTagged`, and fold it into `standard()` and `linterRules()` so every component built on them inherits it: ```jsonnet notOnMirror:: { 'if': '<mirror predicate> && $COMMON_CI_TASKS_VALIDATE_ON_MIRROR != "true"', when: 'never', }, ``` Components that intentionally *should* run on the mirror (release-platform sync, `mirroring`'s `notify_mirror_source`, goreleaser publish on ops) don't use `standard()` today and are unaffected. ### 2. Mirror detection must be explicit There is no predefined CI variable that says "this project is a pull mirror", and `$CI_SERVER_HOST != "gitlab.com"` is **not** a mirror signal: many gl-infra projects are canonical on ops.gitlab.net, and security mirrors live on gitlab.com. An earlier draft of this proposal reused the host-based `GITLAB_INFRA_MIRROR` convention (`$CI_SERVER_HOST == $GITLAB_INFRA_MIRROR`) plus a `security/` namespace regex. IaC planning in gitlab-com/gl-infra/infra-mgmt#26 showed the host convention cannot classify **same-host** mirror pairs: several push-mirror pairs in infra-mgmt are gitlab.com canonical → gitlab.com `security/` mirror (git, gitaly, gitlab-agent, duo-ui). A host-valued variable set on both sides there would make the predicate match on the **canonical** too. Detection is therefore **project-URL-based**, using variables provisioned by IaC on the **mirror project only** (see §6): | Variable (mirror project only) | Meaning | |---|---| | `GITLAB_INFRA_MIRROR_PROJECT_URL` | The mirror project's own URL. A project is a mirror iff `$CI_PROJECT_URL == $GITLAB_INFRA_MIRROR_PROJECT_URL`. Unset ⇒ canonical, the safe failure mode (validation runs). CI `rules:` supports variable-to-variable comparison, so this works for job rules and include rules alike. | | `GITLAB_INFRA_CANONICAL_PROJECT_URL` | The canonical project's URL. Not used for detection; gives `notify_mirror_source` a direct pointer back to the canonical. | In `rules.libsonnet`, with the `security/` namespace regex retained only as a **fallback** for security mirrors not (yet) provisioned through the push-mirror module: ```jsonnet local isManagedMirror = '$GITLAB_INFRA_MIRROR_PROJECT_URL != null && $GITLAB_INFRA_MIRROR_PROJECT_URL != "" && $CI_PROJECT_URL == $GITLAB_INFRA_MIRROR_PROJECT_URL'; local isSecurityMirrorFallback = '$CI_SERVER_HOST == "gitlab.com" && $CI_PROJECT_NAMESPACE =~ /\\/security(\\/|$)/'; local isMirror = '((%s) || (%s))' % [isManagedMirror, isSecurityMirrorFallback]; ``` The fallback expression should be lifted out of `releasePlatform`'s local `onSecurity` so both guards share one definition; it can be dropped entirely once all security mirrors are IaC-managed. Why one URL-based mechanism instead of the previous two-convention OR: 1. **One mechanism** covers infra mirrors *and* security mirrors; the namespace regex is only a transitional fallback. 2. **Chained mirrors** (canonical → security → dev, e.g. git/gitaly) work: with mirror-side-only writes, each project gets the variables from at most one push-mirror module instance, and the middle project is correctly classified as a mirror. Writing a host variable on both sides would have two module instances write the same key with conflicting values on the middle project. 3. **No misclassification** of projects that are canonical on ops.gitlab.net; URLs are unique even on the same host. Cost: this diverges from the legacy `GITLAB_INFRA_MIRROR` host convention that `mirroring`'s `notify_mirror_source` keys off. Since detection isn't implemented yet, now is the right time to switch; the push-mirror module can keep writing the legacy host variable on cross-instance pairs during migration (§6), and `notify_mirror_source` can migrate to `GITLAB_INFRA_CANONICAL_PROJECT_URL` as a follow-up. Neither variable introduces a new consumer-facing input; consumers whose topology fits neither the managed variables nor the fallback are treated as canonical, which is the safe failure mode (validation runs). ### 3. The flag `COMMON_CI_TASKS_VALIDATE_ON_MIRROR=true` as a project-level CI/CD variable on the mirror. Project variables are set by maintainers and are unaffected by `ci_pipeline_variables_minimum_override_role`, which security mirrors use to block pipeline-run variables (that restriction is why release-platform's merge-train uses an input; it does not apply here). Optionally also expose it as a `validate_on_mirror` boolean input on `golang-build` and the other umbrella templates for consumers who want it in-repo. ### 4. Apply the same predicate to includes, not just jobs gitlab-com/gl-infra/tenant-scale/cells-infrastructure/team#846 failed at config assembly. Job rules can't help there. Any `include:component` in our templates that points at a project which may not exist on the mirror (`modular-feature-testing/*`, anything under `gitlab-org/quality/…`) should carry the *same* mirror predicate in its `rules:`, replacing the ad-hoc `$CI_SERVER_HOST == "gitlab.com"` from !1785. That keeps one definition of "mirror" across jobs and includes. A CI `include` whose rule is false is never resolved, so it cannot 404. The URL-based predicate is a variable-to-variable comparison, which `include:rules` supports. ### 5. Scope: which components In scope (currently on `rules.standard` / `linterRules`): `go-unittests`, `golangci-lint`, `go-mod-tidy`, `go-logging-field-validator`, `cargo-{build,clippy,fmt,test}`, `rspec`, `rubocop`, `shellcheck`, `shfmt`, `jsonnetfmt`, `jsonnetlint`, `jsonfmt`, `hclfmt`, `terraform-{format,validate,docs}`, `helm-{lint,unittest}`, `checkov`, `conftest`, `opa`, `promlint`, `protofmt`, `protolint`, `editorconfig-check`, `gitleaks`, `gitlint`, `asdf-tool-versions`, `nix`. Out of scope (side effects or instance-specific by design): `goreleaser`, `semantic-release`, `release-platform`, `mirroring`, `renovate-bot`, `autolabels`, `cng-image-copy`, `api-docs-refresh`, `danger` (needs the canonical MR API anyway), `container-diff` (needs the built image on that instance). Worth deciding explicitly: `goreleaser_validate` is deterministic but sits in the release component; probably follow the rule. ### 6. Provisioning: who sets the detection variables and the flag Today no IaC writes any mirror-detection variable; the legacy `GITLAB_INFRA_MIRROR` exists only where someone set it by hand. Provisioning moves into IaC — the infra-mgmt side is tracked in gitlab-com/gl-infra/infra-mgmt#26 — on the pattern the project module already uses for `AUTOLABELS_ENABLED` (a `gitlab_project_variable`, `raw = true`, described as "(IaC managed)"): | Variable | Owner | Rationale | |---|---|---| | `GITLAB_INFRA_MIRROR_PROJECT_URL` + `GITLAB_INFRA_CANONICAL_PROJECT_URL` (on the **mirror** project **only**) | `project//modules/push-mirror` sub-module | It is the only place that knows which project is canonical (`project_id`) and which is the mirror (`mirror_project`), already has the mirror-side provider alias (`gitlab.mirror`), already manages mirror-side resources (`mirror_branch_protections`, `mirror_tag_protections`), and already resolves the mirror via `data.gitlab_project.mirror` (`web_url`), so the mirror URL is available; the canonical URL needs a data lookup of `project_id` (or a new input). Works for the ownership-scoped pattern (`mirror_project = { id = … }`), the legacy `gitlab-com` / `ops-gitlab-net` environments (`mirror_project = { path = … }`), and same-host security mirrors with no special-casing. Mirror-side-only writes keep chained pairs conflict-free. | | `COMMON_CI_TASKS_VALIDATE_ON_MIRROR` (on the **mirror** project only) | `project` module, new `common_ci_tasks.validate_on_mirror = optional(bool, false)` | Sits next to `common_ci_tasks.mirroring` / `mirroring_remote_instance`, which the mirror-side module block already declares (see infra-mgmt `HOWTO.md`, "Mirror project"). Same shape as `autolabels` → `AUTOLABELS_ENABLED`. | **Migration compatibility:** the sub-module can optionally keep writing the legacy `GITLAB_INFRA_MIRROR` (mirror host) on both sides of *cross-instance* pairs, behind a toggle, while `notify_mirror_source` still keys off it; removed once the `mirroring` component migrates to `GITLAB_INFRA_CANONICAL_PROJECT_URL`. On the infra-mgmt side (gitlab-com/gl-infra/infra-mgmt#26): bump the `project` module across environments (via Renovate), audit every `push-mirror` usage (`environments/chef/`, `dedicated/`, `delivery/`, `gitlab-com/`, `ops-gitlab-net/`) including the chained pairs, and reconcile pre-existing hand-set `GITLAB_INFRA_MIRROR`-style variables (Terraform cannot create a variable that already exists: delete or `import` first). Consumers then opt in per project in their infra-mgmt environment `.tf`, not by hand in the UI. The `common_ci_tasks.mirroring = true` blocks across infra-mgmt environments also give the audit in *Rollout* step 2 its population: every project module with `mirroring = true` is either a canonical or a mirror, and the paired `push-mirror` module says which. Security mirrors provisioned through push-mirror pairs need no special casing; the namespace-regex fallback covers only unmanaged ones. ## Why an opt-in and not a hard rule `mirroring.md` documents that "many Infrastructure projects are hosted on one GitLab instance (usually GitLab.com), but run pipelines from another (usually ops.gitlab.net)". For those projects the ops "mirror" is the **only** place validation runs, and `notify_mirror_source` posts results back to the gitlab.com MR. Flipping the default without an audit would silently remove their validation. Hence: - the flag exists, - the default flip ships in a **major version** with a migration note, and - the rollout starts opt-in. ## Rollout 1. **v5.x, additive:** land `rules.notOnMirror` (URL-based predicate + namespace fallback), the include-level predicate, docs, and manitest coverage, gated behind `COMMON_CI_TASKS_SKIP_VALIDATION_ON_MIRROR=true` (opt-in to skipping). Migrate the !1785 `go-test` gate onto the shared predicate. 2. **IaC + audit (gitlab-com/gl-infra/infra-mgmt#26):** in `terraform-modules/gitlab/project`, have `push-mirror` write `GITLAB_INFRA_MIRROR_PROJECT_URL` and `GITLAB_INFRA_CANONICAL_PROJECT_URL` on the mirror project only, and add `common_ci_tasks.validate_on_mirror`; decide legacy `GITLAB_INFRA_MIRROR` compatibility; bump infra-mgmt to the new module version and reconcile pre-existing manual variables. Enumerate every project module in infra-mgmt with `common_ci_tasks.mirroring = true`, classify each mirror as "validation runs on canonical" vs "mirror is the only validation", and set `validate_on_mirror = true` on the latter. 3. **v6.0:** invert the default; drop the skip flag; keep `COMMON_CI_TASKS_VALIDATE_ON_MIRROR`. Renovate will surface the bump to every consumer with the changelog note. 4. Consumers remove their local workarounds (gitlab-com/gl-infra/terra-transformer!906, gitlab-com/gl-infra/reaper!11, gitlab-com/gl-infra/platform/runway/runwayctl!2002). ## Cost note (keep expectations honest) `rules.standard` already restricts validation to MR pipelines and the default branch. Ops mirrors generally have no MRs, so the duplicated spend today is mostly **default-branch pushes** (and tag pipelines for components with `allowTags`). Real, but the fragility argument is the stronger one; a before/after count of validation jobs on ops per week would quantify it. ## Acceptance criteria - [ ] `rules.libsonnet` exposes a single `isMirror` predicate (`$CI_PROJECT_URL == $GITLAB_INFRA_MIRROR_PROJECT_URL`, with the `security/` namespace regex as a fallback for unmanaged security mirrors) used by both job rules and template includes, with `releasePlatform`'s `onSecurity` sharing the namespace definition - [ ] `standard()` / `linterRules()` skip on mirrors when the flag says so; manitests cover canonical, managed mirror (URL match), unmanaged security mirror (namespace fallback), and flag-set cases - [ ] The `go-test` include in `golang.jsonnet` uses the shared predicate instead of `$CI_SERVER_HOST == "gitlab.com"` - [ ] `README`/`mirroring.md` document the rule, the detection mechanism, and the flag - [ ] `push-mirror` sub-module provisions `GITLAB_INFRA_MIRROR_PROJECT_URL` and `GITLAB_INFRA_CANONICAL_PROJECT_URL` on the mirror project only, for all pair patterns (cross-instance, same-host security, chained); decision recorded on legacy `GITLAB_INFRA_MIRROR` compatibility; `project` module exposes `common_ci_tasks.validate_on_mirror` → `COMMON_CI_TASKS_VALIDATE_ON_MIRROR` (gitlab-com/gl-infra/infra-mgmt#26) - [ ] infra-mgmt bumped to the new module version with pre-existing manual variables reconciled; every `mirroring = true` project audited and `validate_on_mirror` set where the mirror is the only validation (gitlab-com/gl-infra/infra-mgmt#26) - [ ] Default inverted in a major release; changelog explains the migration - [ ] terra-transformer, reaper, runwayctl workarounds removed ## Open questions - Should the ops-mirror **Renovate** be disabled or restricted for mirrored projects? gitlab-com/gl-infra/tenant-scale/cells-infrastructure/team#846 was partly caused by Renovate on the mirror bumping `common-ci-tasks` past a version that couldn't resolve there; validation-skipping doesn't fix that. - Do we want a **canary** that runs `gitlab-ci lint` against the mirror's merged config on each `common-ci-tasks` release, so config-assembly breakage on ops is caught before consumers hit it? - When should `mirroring`'s `notify_mirror_source` migrate from the legacy `GITLAB_INFRA_MIRROR` host variable to `GITLAB_INFRA_CANONICAL_PROJECT_URL`, so the legacy variable (and the push-mirror compatibility toggle) can be dropped? ## References - gitlab-com/gl-infra/infra-mgmt#26 (IaC provisioning of the mirror-detection variables; source of the project-URL-based detection design) - gitlab-com/gl-infra/tenant-scale/cells-infrastructure/team#846 (root-cause writeup) - gitlab-com/gl-infra/common-ci-tasks!1785 (bespoke go-test gate, merged 2026-09-16) - gitlab-com/gl-infra/common-ci-tasks!1734 (made the go-test include unconditional) - gitlab-com/gl-infra/terra-transformer!906, gitlab-com/gl-infra/reaper!11, gitlab-com/gl-infra/platform/runway/runwayctl!2002 (consumer-side workarounds) - gitlab-com/gl-infra/delivery#21789 (Implement mirror-aware CI jobs, Release Platform precedent) - `src/ci/lib/rules.libsonnet` (`standard`, `linterRules`, `releasePlatform`), `src/ci/components/mirroring.jsonnet` (legacy `GITLAB_INFRA_MIRROR` convention) - gitlab-com/gl-infra/infra-mgmt `HOWTO.md` → "Mirroring projects between instances" (canonical / mirror / push-mirror module pattern) - gitlab-com/gl-infra/terraform-modules/gitlab/project `common-ci-tasks.tf` (`AUTOLABELS_ENABLED` IaC-managed variable precedent), `modules/push-mirror`