Build gitlab-rails before waiting for the assets image

Heinrich Lee Yurequested to merge
split-rails-assets-build into master

What does this MR do?

Splits the Debian gitlab-rails-* build so that only a short final step needs the assets image, when that image doesn't exist yet. This is step 2 of #2441.

Most of the rails build (bundle install, gettext:compile, bootsnap) doesn't use the assets. But the assets image is a FROM stage, so none of it can start until that image exists. Since gitlab-org/gitlab!263017 (merged), E2E on CNG starts gitlab-rails-ee before the assets image is built, so in cold MR pipelines the job waits for it. With this MR, that work overlaps with the wait.

In paired test pipelines (below), the split got the rails job done 1.9–3.6 minutes earlier in pairs 1, 3, 5 and 6, about the time the control spent waiting. The split adds no measurable build time. In production, cold MR pipelines wait a median of 2 minutes (90th percentile 3.5, longest 13.5), so I expect about 2 minutes saved per cold pipeline, and up to about 6 minutes when the assets build is slow.

gitlab-rails/Dockerfile.erb

  • The runtime stage is now named rails-no-assets. It has every runtime layer except the assets.

  • The assets stage is declared at the end and used only by a new last stage, final, which is still the default target:

    FROM rails-no-assets AS final
COPY --chown=git --from=assets --exclude=webpack/**/*.map assets /srv/gitlab/public/assets/

  • --exclude replaces the old find … -name '*.map' -delete, so the maps are never added to a layer. With COMPILE_ASSETS=true the find stays in builder and there's no assets stage.

  • The published image has the same files, ownership and modes. The assets are now their own last layer instead of being inside the /srv/gitlab copy.

build-scripts/build.sh

  • build_target_only <stage> [args] runs build_if_needed with --target. It skips the tag, push, --metadata-file, zstd output, skopeo copy, signing and artifact files.
    • It keeps the same --cache-to/--cache-from refs, so the final build reuses its layers from the sticky BuildKit node or from the registry cache.
    • It skips the build when needs_build is false.
  • rails_split_assets_build reads RAILS_SPLIT_ASSETS_BUILD:
    • It is off by default. RAILS_SPLIT_ASSETS_BUILD=true turns it on.
    • It always stays off for UBI, FIPS, nix, COMPILE_ASSETS, stable branches, tags and auto-deploy, and when the build cache is disabled (DISABLE_DOCKER_BUILD_CACHE, CACHE_DOCKER_BUILD=false). Phase B only reuses Phase A through the cache.
  • rails_split_assets_build_now is rails_split_assets_build, but only while the assets image is missing (assets_image_available from !3173 (merged)). Once the image exists, a single build is faster, since there's nothing left to overlap. It logs which path it takes.
  • build_if_needed no longer copies shared/ to shared/shared on a second build in the same job.

.gitlab/ci/common.gitlab-ci.yml (.gitlab-rails-ee, .gitlab-rails-ce)

  • The build args are collected once in rails_build_args, so both phases use exactly the same args, which keeps the cache hits.
  • When the split is on: download_gitlab, then the image check. If the image is still missing: Phase A (build_target_only rails-no-assets). Then fetch_assets, Phase B (build_final_image), push_tags and the vendor bundle extraction.
    • The check runs after download_gitlab, which doesn't need the assets, so the image has the most time to appear. In one test run the download took more than 6 minutes and the image appeared meanwhile, so the job built in one step.
  • When it's off, the order is the same as today: fetch_assets first, then the single build. UBI/FIPS keep fetch_assets before build_asset_image.
  • ci_files/final_images.yml and the downstream images are unchanged. Webservice, sidekiq, toolbox and workhorse still use the full rails image.

When to turn it on

The split only saves time when gitlab-rails-* starts before the assets image exists, for example in pipelines that gitlab-org/gitlab!263017 (merged) triggers as soon as the assets hash is known. It's opt-in: a pipeline that wants it passes RAILS_SPLIT_ASSETS_BUILD=true. gitlab-org/gitlab already sends true from E2E on CNG.

The wait-for-assets-* gate from !3173 (merged) skips pipelines that set RAILS_SPLIT_ASSETS_BUILD=true, so the rails job can start early.

How was it tested?

Paired pipelines

Each pair ran two gitlab-org/gitlab MR pipelines at the same time, each with a cold assets compile (a different comment in main.js). Run A used this branch, run B used the CNG master commit it's based on (8b57dab5), and both set RAILS_SPLIT_ASSETS_BUILD=true. Times are seconds within gitlab-rails-ee, and minutes from pipeline start for when it finished.

Pair Path in A B waited for the image A: Phase A + Phase B B: single build Rails job done, A / B
1 split 180s 618 + 55 = 674s 682s 18.1 / 21.6 min (−3.5)
2 split, checked at job start (old version) 240s 413 + 63 = 476s 386s 21.1 / 18.6 min (+2.5)
3 split 150s 385 + 63 = 448s 439s 15.3 / 17.2 min (−1.9)
4 one step: the image existed after a 387s download 690s 422s (single) 451s not comparable: B's assets image was 9 min late
5 split 270s 330 + 53 = 382s 454s 15.9 / 19.5 min (−3.6, flatters A: B's assets image was 1.9 min later)
6 split 210s 399 + 61 = 460s 432s 14.2 / 17.5 min (−3.3)
  • A split job never waited for the image: Phase A always outlasted the wait.
  • Phase B takes 53–63 seconds, but Phase A is shorter than a full single build. The split's total build time differed from B's single build by −72 to +90 seconds, about +10 on average, which is within the run-to-run noise.
  • So the split saves about what B spent waiting.
  • Pair 2 used the earlier version, which checked for the image at the start of the job. Its download took 319 seconds, the image appeared meanwhile, and the job paid for the split without overlapping anything. That's why the check now runs after download_gitlab, which pair 4 confirms.
  • With the image already there (a warm pipeline), the job builds in one step, as before.

Image

  • In pair 2, E2E on CNG deployed the split image with the Helm chart and passed: 26 of 27 jobs, the 27th failed on a DNS error in the test runner before any test ran. Webservice, sidekiq, toolbox and workhorse built on top of it.

Local and unit tests

I used BuildKit moby/buildkit:v0.26.3, the same as BUILDKIT_IMAGE in CI:

  • --target rails-no-assets succeeds when ASSETS_IMAGE doesn't exist. BuildKit doesn't resolve the unused stage, so no fallback Dockerfile is needed.
  • COPY --exclude=webpack/**/*.map works with the built-in frontend (no # syntax= line). It removes maps at every depth under webpack/ and keeps other files.
  • In a second build with the default target, every Phase A step is CACHED. Only the assets COPY runs.
  • buildx build --check on the rendered EE and COMPILE_ASSETS Dockerfiles gives the same warnings as master.
  • build-scripts/tests/rails_split_assets_build.bats covers the opt-in, the builds that never split, and that the split only happens while the assets image is missing (stubbed skopeo).

Still to check:

Related

Edited by Heinrich Lee Yu

Merge request reports