Gate gitlab-rails-* on the upstream assets image

What

Adds the wait-for-assets-{ce,ee} gate jobs (stage prepare on both Debian and UBI, needs: [], using a plain runner without dind/buildx). gitlab-rails-{ce,ee} get an optional needs: on the gate, so they wait for it to clear before starting. The gate polls the registry for the upstream gitlab-assets-{ce,ee}:${GITLAB_ASSETS_TAG} image via a new wait_for_assets helper.

The rails jobs now call CONTAINER_VERSION=$(get_rails_version) fetch_assets, so fetch_assets sees the rails version while build_asset_image does not.

Related to team-tasks#274 and the incident gitlab-com/gl-infra/production#23186 (The 19.5 Release was blocked for approximately 7hrs).

Why

Findings from the investigation in team-tasks#274:

1. No ordering guarantee: The assets image is built by the upstream GitLab pipeline (cache:assets-hash → compile-production-assets → build-assets-image). Upstream-triggered CNG pipelines are gated on build-assets-image in the Upstream Rails CI, but CNG-initiated pipelines (master pushes, MRs, their ubi/fips downstreams) are not. fetch_assets is polled inside gitlab-rails-* as a fetch-sleep loop.
2. The poll ran on the most expensive runner: gitlab-rails-* uses the ${HIGH_CAPACITY_RUNNER_TAG}, which starts the dind/buildx builder, then polls. With .job-base having retry: 1, that means two jobs per pipeline, and every phase-six image blocked behind them. fetch_assets now has a 45 min limit (!3179 (merged), on master), which bounds this but still spends that time on the high-capacity runner.
3. The rails jobs wait for an image that is not yet in the registry: fetch_assets polls skopeo inspect for the upstream gitlab-assets-{ce,ee} image, and in team-tasks#274 that image had not been built yet when gitlab-rails-ee/gitlab-rails-ce started. It is a transient condition, but it has failed pipelines during releases, so this is intended as a stopgap rather than outright changing how CNG sources its assets image. This is a small gate on a plain runner that checks whether the assets image has been built before gitlab-rails-* claims its runner, saving that runner time and cost.

How it works

• Skips the wait if COMPILE_ASSETS=true.
• Probes the repository once via curl (tags-list, n=1); a wrong image path (404 NAME_UNKNOWN) fails immediately with exit 1, which is not retried.
• Polls only while skopeo reports a missing manifest.
• Transient errors are retried ASSETS_WAIT_TRANSIENT_RETRIES times (default 3) with exponential backoff (interval * 2^(failures-1), i.e. 30s, 60s, 120s). Transient means transport errors (DNS, connection, TLS, timeouts), registry 5xx (received unexpected HTTP status: 500/502/503/504/429). The 5xx wording is inferred from skopeo's error format, not captured from a real failure; the patterns are one list in assets_image_last_error_is_transient so they are easy to extend.
• Any other error is passed through as skopeo reported it, with skopeo's exit status intact, and is not retried.
• If the assets are still missing after ASSETS_WAIT_TIMEOUT the gate returns exit code 75, and the job retries only on exit code 75 (one retry).

Timeouts and retry arithmetic

• Hosted GitLab.com runners stop jobs at 3h regardless of the job timeout, so the gate has timeout: 3h and ASSETS_WAIT_TIMEOUT defaults to 9900s (2h45m), leaving headroom for start-up and the final poll so the gate, not the runner, reports the failure.
• 2h45m per attempt, one retry on exit 75: about 5.5h in total before the pipeline fails.
• ASSETS_WAIT_TIMEOUT is not set at job level on purpose: a job-level variables: entry would shadow project/pipeline variables, so it could no longer be tuned without a code change. The default lives in wait_for_assets.

Testing

• bats --print-output-on-failure build-scripts/tests: all 107 tests pass locally, including 19 in wait_for_assets.bats with stubbed skopeo and curl.
  • Covered: COMPILE_ASSETS skip, repository probe (unknown repo, inconclusive probe, no job token, registry port), poll loop, job token credential, transient retries (network, 503 that recovers, toomanyrequests), exponential backoff (5/10/20s with a stubbed sleep) vs. flat polling for a missing manifest, error pass-through, exit 75 at the deadline.
• shellcheck on the test stubs and wait_for_assets.bats is clean; the build.sh shellcheck warning count is unchanged from master.
• Please play the manual fips_pipeline bridge to validate UBI/FIPS.

Follow-ups

• Move GITLAB_ASSETS_TAG off the floating ref slug onto the immutable assets-hash-<sha> tag for branch pipelines (same hash across .com, dev and the security mirror; tag pipelines keep the tag name).
• Check the upstream registry cleanup policies keep master and *-stable-ee.
• Optionally fall back to COMPILE_ASSETS=true after the deadline for a deterministic build.