Start E2E on CNG before the assets image is built

Heinrich Lee Yurequested to merge
cng-2441-start-e2e-cng-earlier into master

What does this MR do?

Starts e2e:test-on-cng as soon as the assets hash is known, instead of after compile-production-assets and build-assets-image. CNG's fetch_assets already waits until the assets image exists, so the CNG pipeline can start while the assets are still compiling. In test runs, the CNG tests started about 5.5–6 minutes earlier.

Changes:

  • .gitlab/ci/caching.gitlab-ci.yml: cache:assets-hash also outputs GLCI_ASSETS_HASH_IMAGE_TAG (assets-hash- plus the first 10 characters of the hash, like assets_image_tag in scripts/utils.sh). It has its own name so it can't clash with the GLCI_ASSETS_IMAGE_TAG the asset jobs output, which differs for tags, Vue 3 and istanbul.
  • .gitlab/ci/qa.gitlab-ci.yml: e2e:test-on-cng needs cache:assets-hash instead of compile-production-assets and build-assets-image, and passes GLCI_ASSETS_IMAGE_TAG: $GLCI_ASSETS_HASH_IMAGE_TAG to the child pipeline.

If an image for the assets hash already exists (warm pipelines), CNG uses it straight away and the asset jobs are no longer on the E2E path at all.

Step 1 of gitlab-org/build/CNG#2441.

Expected savings and cost

Cold (assets compiled) Warm (assets image already exists) Average
Share of E2E pipelines 75% (94 of 125) 25% (31 of 125)
CNG tests start earlier by about 5.5 min about 6 min about 5.6 min
gitlab-rails-ee waiting in fetch_assets about 8 min 0 about 6 min
Extra CNG runner time about 7.5–8 min 0 about 6 min
  • The shares come from successful MR pipelines with e2e:test-on-cng on 2026-10-16, 08:00–20:00 UTC, split by whether compile-production-assets compiled.
  • The wait is the time between gitlab-rails-ee starting to poll in fetch_assets (about 8 minutes into the pipeline) and this pipeline pushing the assets image (median 15.1 minutes in cold pipelines). In the cold run it polled 16 times, every 30 seconds.
  • The extra runner time is that wait: gitlab-rails-ee took 1166s in the cold run, against 720s in the warm run and 654s in an earlier pipeline without this change. No other job runs longer. The trigger jobs don't use runners, and the CNG test jobs start earlier but take as long as before.
  • !263020 (merged) makes e2e-test-pipeline-generate, which this trigger waits for, about a minute faster. With both merged, E2E starts about a minute earlier again in warm pipelines, and cold pipelines wait about a minute longer in fetch_assets.
  • gitlab-org/build/CNG#2441 (step 2) would use the waiting time for the Rails build instead.
Measurements

Time from pipeline start until the first CNG test job starts:

Without this change: median of MR pipelines on 2026-10-16, 08:00–20:00 UTC With this change (one run each) Difference
Warm (assets cached) 26.0 min (31 pipelines) 20.0 min (https://gitlab.com/gitlab-org/gitlab/-/pipelines/2957860637) about −6 min
Cold (assets compiled) 35.3 min (94 pipelines) 29.8 min (https://gitlab.com/gitlab-org/gitlab/-/pipelines/2957945860) about −5.5 min

Cold run timeline (a temporary commit, since dropped, forced a new assets hash):

Time (UTC) From start What happens
05:40:33 0 pipeline created
05:43:13 +2.7 min e2e:test-on-cng starts
05:45:58 +5.4 min build-cng starts
05:48:34 +8.0 min gitlab-rails-ee starts waiting in fetch_assets
05:56:21 +15.8 min build-assets-image pushes the image
06:06:13 +25.7 min gitlab-rails-ee finishes
06:10:23 +29.8 min first CNG test job starts

In the warm run, fetch_assets found the image immediately, about 1.5 minutes before this pipeline's own build-assets-image finished.

The test runs were on a Saturday morning (UTC), the baseline on a busy Friday. The comparison is one run each against medians.

Before merging

  • Merge gitlab-org/build/CNG!3179 (merged) first. Without it, fetch_assets has no time limit: if the asset jobs fail, gitlab-rails-ee keeps waiting until its job timeout, and this pipeline stays running. With it, the job fails after ASSETS_IMAGE_WAIT_TIMEOUT_SECONDS (45 minutes by default).
  • While it waits, gitlab-rails-ee holds a CNG runner (about 8 minutes in the cold run). See "Expected savings and cost" above.

Related issues

Checklist

Pre-merge

Consider the effect of the changes in this merge request on the following:

  • Different pipeline types: only e2e:test-on-cng changes. Check the other pipeline types that run it.
  • Non-canonical projects:
    • gitlab-foss
    • security
    • dev
    • personal forks
  • Pipeline performance: CNG tests start about 5.5–6 minutes earlier in the runs above.

If new jobs are added:

This will help keep track of expected pipeline time and cost increases.

Post-merge

Edited by Heinrich Lee Yu

Merge request reports