Draft: Gate -nix pushes on asset name parity

Alessio Caiazzarequested to merge
nolith-uwlzrmqzqyom into master

What does this MR do?

It adds a gate that stops a -nix image from being pushed when its frontend assets name a file that the Debian assets image does not have.

What breaks without it

  • GitLab.com serves /assets from a bucket, not from the pod. The deployer fills that bucket from the Debian gitlab-assets-ee image (ADR-0045, "Context and Problem Statement").
  • Rails on a Nix pod writes the asset names of its own build into each page (lib/gitlab/webpack/manifest.rb).
  • When the bucket lacks one of those names, the browser gets a 404. The staging canary on Nix images served almost blank pages that way (production#23043).

The hardened-builds fixes make the Nix names match upstream's. They are all merged and in the current pin (listed under "Related issues"). Nothing checks that they stay that way. Without the gate, the first sign of a new difference would be 404s on GitLab.com.

What the gate does

Before the first -nix image with the assets is pushed, the gate:

  1. takes the webpack, Rspack and Sprockets manifests of the Nix assets build;
  2. streams the same manifests out of the Debian assets image for the same GitLab commit (crane export, about 230 MB for EE);
  3. fails when the Nix manifests name a file that the Debian image does not have.

Source maps and manifest metadata do not count. Files that only the Debian image has are fine. Each side needs at least one webpack manifest: manifest.json up to 19.5, and only manifest.rspack.json from 19.6 on, because GitLab now builds its production assets with Rspack only (gitlab-org/gitlab!263628 (merged)). Each webpack manifest must be on both sides or on neither. The details are in docs/nix-variant.md, "Asset parity with the Debian variant".

Scope. The gate checks the direction that breaks GitLab.com today: every name a Nix pod writes into a page must be in the bucket. ADR-0045's decision asks for more: the same file names on both sides, and every file upstream's image ships, source maps included, in the Nix output. The gate does not check that reverse direction yet. That is left for a follow-up.

Piece Change
gitlab-assets-{ce,ee} Save their manifests and the locked gitlab-rails commit as artifacts.
build-scripts/nix_assets_parity.sh New. Finds the Debian image, streams its manifests and compares.
gitlab-assets-parity-{ce,ee} New jobs. They run next to gitlab-rails-{ce,ee}.
gitlab-rails-{ce,ee} Run the same check during their build, and push only when it passes.
webservice, sidekiq, toolbox, workhorse, geo-logcursor Need the parity job of their edition before they push.
rails closure cache Also uploads when the job fails, but only a completed export.

Why the rails jobs run the check themselves: each image job builds and pushes in one go. A needs: on the parity job would hold back the rails build, the longest step of the pipeline. So the rails jobs run the check in the background and push nothing until it passes. No build waits for the gate.

Why the rails cache uploads on failure: a failed check fails gitlab-rails-* after its whole build. With the closure in the cache, a retry does not rebuild it. An after_script guard (nix_cache_guard) keeps the cache only when the closure export completed, so a partial export is never uploaded (docs/nix-variant.md, "Where the gate sits").

Exit Meaning Job result
0 Every file the Nix manifests name is in the Debian image. passes
1 A file, an entrypoint or a manifest differs. fails
2 The gate cannot check: a manifest, the image, a tool or a variable is missing. fails
3 Branch pipelines only: upstream built no assets image for the locked commit. warning

Which Debian image the gate compares with:

  • Tag and auto-deploy pipelines: the image the Debian jobs copy (GITLAB_ASSETS_TAG). The gate waits up to 30 minutes for it, then fails with exit 2.
  • Branch pipelines (MRs, master): GITLAB_ASSETS_TAG names a branch there, and that tag moves. So the gate takes the image of the commit the Nix registry locked. Upstream builds that image only for some commits, and gitlab-foss images carry gitlab-foss commits, so CE always exits 3 here.
  • Auto-deploy builds no CE (.nix-ce-rules), so only tag pipelines check CE.

On a mismatch, the job log lists a few of the differing names. The full lists are in the job artifacts, under artifacts/nix-assets-parity/. On exit 3, no-reference.txt there says why there was nothing to compare with.

How to test

  • Locally: run bats build-scripts/tests with bash 5 and jq. GNU and BusyBox tar and awk both work. Then run shellcheck on the file list of lint:build-scripts-shellcheck (.gitlab/ci/tests.gitlab-ci.yml).
  • In CI: in an MR pipeline, start the manual nix_pipeline job. In its child pipeline, expect gitlab-assets-parity-ee to pass and gitlab-assets-parity-ce to warn with exit 3. The gitlab-rails-{ce,ee} logs show the same check.

Before merging

Nothing outside this MR blocks it any more. Every hardened-builds fix is merged and in the pin. gitlab-org/gitlab!262551 (merged) made the Rspack names independent of the checkout path and, like gitlab-org/gitlab!261541 (merged), is backported (ADR-0045, "What stays intact").

One cost to know about: the parity jobs have no cache, so nix_setup builds the hardened-builds ci-toolchain (Go, cosign, skopeo, the Yarn 4 fetcher) in each of them. With the current pin that took 27m40s in gitlab-assets-parity-ee, out of the job's 50 minute script limit. The check itself took 47 seconds. Other nix jobs pay the same setup today (26m20s in certificates). If it grows, the toolchain needs a cache, or .nix-assets-parity a longer RUNNER_SCRIPT_TIMEOUT.

Related issues

The hardened-builds fixes split out of hardened-builds !230:

MR Change State
!235 Drop the obsolete Vue compiler workarounds merged, in the current pin
!236 Run @gitlab/web-ide's postinstall merged, in the current pin
!238 Ship precompressed .gz assets merged, in the current pin
!239 Build assets with source maps merged, in the current pin
!240 Install only production JS dependencies merged, in the current pin
!241 Check dependency install scripts against a list merged, in the current pin
!242 Compile from GitLab's own yarn.lock merged, in the current pin

The current pin is hardened-builds d25abede (ci_files/variables.yml, 2026-10-29).

Test plan

At d286c440, rebased onto master 55409ac0:

  • bats, full suite: 189/189 under bash 5.3.20, GNU tar 1.35 and GNU Awk 5.4.1, with 1 skip (the real-data test below, which needs local trees). This MR adds nix_assets_parity.bats (64 tests) and nix_cache_guard.bats (10 tests). Their fixtures are cut from the real 19.2.1 manifests. They cover a renamed entrypoint, lazy and worker chunks, a missing entrypoint, a webpack manifest on one side only, the Rspack-only manifests of 19.6, Sprockets mismatches, bad input, the exit-3 report, and the runner's set -eo pipefail. The 7 tests added or changed for 19.6 fail against the previous version of the gate.
  • shellcheck 0.11.0 on the lint:build-scripts-shellcheck file list: clean.
  • BusyBox tar and awk (the bats/bats:1.11.0 image): 189/189, 1 skip, in test:build-scripts.
  • Real data, 19.6 EE, outside CI: the manifests that Nix run 3003802259 handed over and fetched for gitlab master 7666d5af. The previous version of the gate exits 2 on them, this one exits 0: manifest.rspack.json 2115 files and 1286 entrypoints, Sprockets 1239 assets, on both sides.
  • Real data, 19.2.1 EE, outside CI, at 41cf0720, before the 19.6 change: the Nix gitlab-assets-ee output and registry.gitlab.com/gitlab-org/gitlab/gitlab-assets-ee:v19.2.1-ee name the same 3485 webpack files, 589 entrypoints and 650 Sprockets assets. Both trees carry manifest.json, which the 19.6 change compares as before. Earlier runs also streamed the image live, and renaming the main chunk made the gate fail.
  • 19.5 with Rspack, outside CI: assets_manifest_compare on a Nix build of the canary auto-deploy 19.5.202610091422 (f2f37752ef9) against dev.gitlab.org:5005/gitlab/gitlab-ee/gitlab-assets-ee:f2f37752ef9e6db6db87a8c72844c4c22dfce858: webpack/manifest.json 4574 files and 921 entrypoints OK, webpack/manifest.rspack.json 1663 files and 921 entrypoints OK, Sprockets 991 assets OK, exit 0. 6 Rspack entrypoints list the same files in another order; the gate compares sets, so that passes.
  • GitLab CI lint and a green MR pipeline on GitLab.com: 3004361718.
  • Nix run 3004364043, at d286c440, building gitlab master 90551fac (19.6.0-prerelease) with the hardened-builds pin d25abede:
  • Nix run 3003802259, at 089e8dd2, before the 19.6 change: gitlab-rails-ee and gitlab-assets-parity-ee failed closed with exit 2 on gitlab master 7666d5af, because neither side had webpack/manifest.json, while every Rspack and Sprockets name matched. That is what the 19.6 change fixes.
  • Nix run 2943419417, on an earlier version of this MR (075840db), building gitlab master 25876b0b with a hardened-builds pin that had none of the fixes:
    • gitlab-assets-parity-ee fails with exit 1, as it should: 1978 webpack files only the Nix build has, and manifest.rspack.json only on the Debian side.
    • gitlab-rails-ee runs the check during its build, fails with exit 1 after the closure export, pushes nothing, and still uploads its closure cache.
    • Not covered by a Nix run since then: a mismatch (covered by bats), and the guard's discard path (a failed build without an export, covered by bats).

Checklist

See Definition of done.

For anything in this list which will not be completed, please provide a reason in the MR discussion

Required

  • Merge Request Title, and Description are up to date, accurate, and descriptive
  • MR targeting the appropriate branch
  • MR has a green pipeline on GitLab.com
  • When ready for review, MR is labeled "~workflow::ready for review" per the Distribution MR workflow

Expected (please provide an explanation if not completing)

  • Test plan indicating conditions for success has been posted and passes (above)
  • Documentation created/updated (docs/nix-variant.md, docs/ci-variables.md, build-scripts/tests/README.md)
  • Integration tests added to GitLab QA: not applicable, CI-only change
  • The impact any change in container size has should be evaluated: none, no image content changes
  • New dependencies are managed with GitLab forked renovatebot: none added; crane and jq come from the hardened-builds ci-toolchain
Edited by Alessio Caiazza

Merge request reports