## Problem In `gitlab-org/gitlab` MR pipelines, `e2e:test-on-cng` is usually the last thing to finish. It only triggers the CNG build once the assets image exists, and then `gitlab-rails-ee` spends most of its time on work that doesn't need the assets. So that work waits behind the frontend compile instead of running alongside it. We'd like to start the CNG build earlier and only make the assets step wait. That could take roughly 3–11 minutes off the E2E path in MRs, depending on how much of it we do. ## Where the time goes In this cold-compile MR pipeline (https://gitlab.com/gitlab-org/gitlab/-/pipelines/2878007956): | Time (UTC) | What happens | |---|---| | 08:47–08:57 | `compile-production-assets` (cold Webpack compile) | | 08:57–09:00 | `build-assets-image` pushes `gitlab-assets-ee:assets-hash-…` | | 09:00–09:02 | `e2e:test-on-cng` child pipeline, `build-cng-env`, `build-cng` trigger | | 09:02–09:03 | `gitlab-rails-ee` job starts (executor setup) | | 09:03:45 | `gitlab-rails-ee` runs `fetch_assets` | | 09:04–09:12 | `gitlab-rails-ee` builds the image (about 8.5 min) | | 09:12–09:17 | webservice, sidekiq, toolbox, workhorse images | | 09:17–09:36 | E2E tests | CNG pipeline: https://gitlab.com/gitlab-org/build/CNG-mirror/-/pipelines/2878060654 <details><summary>Why most of the Rails build doesn't need the assets</summary> - `gitlab-rails/Dockerfile.erb` copies the assets near the end of the builder stage (`COPY --chown=git --from=assets assets /srv/gitlab/public/assets/`): https://gitlab.com/gitlab-org/build/CNG/-/blob/master/gitlab-rails/Dockerfile.erb - Before that, it runs `yarn install`, `bundle install`, `gettext:compile` and generates the bootsnap cache. None of these use the assets, and together they're most of the job's time. - But the assets image is a stage (`FROM ${ASSETS_IMAGE} AS assets`), and `fetch_assets` waits for it before the build starts. So all of that work only starts once the assets exist. </details> ## Proposal ### Step 1: trigger CNG earlier (change in `gitlab-org/gitlab`) `fetch_assets` in `build-scripts/build.sh` already waits until the assets image exists, checking every 30 seconds: https://gitlab.com/gitlab-org/build/CNG/-/blob/master/build-scripts/build.sh So `e2e:test-on-cng` could start as soon as `cache:assets-hash` has computed the assets hash, instead of after `build-assets-image`. Expected saving: about 3 minutes. That's the time between the assets image being pushed and `gitlab-rails-ee` reaching `fetch_assets`. What CNG would need to support this: - **Check the image's contents, not just that it exists.** Today, waiting for `build-assets-image` also stops CNG from using an old image under the same tag, such as one built before Rspack assets were added. If CNG starts first, `fetch_assets` would accept any image with that tag. https://gitlab.com/gitlab-org/gitlab/-/merge_requests/260188 adds a label to the assets image, `org.gitlab.assets-image.layout=webpack-rspack`. `fetch_assets` could wait until the image has the expected label. - **Stop waiting at some point.** If the frontend compile fails, `fetch_assets` would wait forever. It needs a time limit, or the GitLab pipeline needs to cancel the CNG pipeline. ### Step 2: build Rails before the assets exist (change in CNG) Split the `gitlab-rails` build so that only a short final step needs the assets image. For example: 1. build everything up to and including the bootsnap cache without the assets stage; 2. a final step, or a separate small job, adds `public/assets` from the assets image once it exists. Then the roughly 8.5-minute Rails build runs while the frontend assets compile. In the pipeline above, that's up to about 8 more minutes off the E2E path. For pipelines where the assets are already cached, the compile is short, so the saving is smaller. ## Questions for Distribution - Is splitting the `gitlab-rails` build like this feasible? Are there other images or pipelines (Omnibus, nightly, release builds) that rely on the assets being part of the same build? - Would you prefer the label check in `fetch_assets`, or a different way for GitLab to say which image to use? - Any concerns about CNG pipelines waiting for an image for longer? ## Related - https://gitlab.com/gitlab-org/gitlab/-/work_items/630697 (asset compilation time in GitLab MR pipelines) - https://gitlab.com/groups/gitlab-org/-/work_items/22918 (pipeline duration epic)