Reuse wait_for_assets in fetch_assets

Heinrich Lee Yurequested to merge
fetch-assets-fail-fast into master

What does this MR do?

Makes fetch_assets in the gitlab-rails-* jobs use wait_for_assets, the same wait the wait-for-assets-* gate runs. This is the follow-up to !3179 (comment 4041741521).

Today fetch_assets treats every skopeo failure as "not available yet" and keeps polling until its 45 min timeout. Auth, registry and usage errors can't clear up during the job, so the polling only hides them. That's how an unsupported --no-tags flag went unnoticed for the whole timeout in !3179 (merged)'s first pipeline.

fetch_assets still matters after !3173 (merged). The gate is skipped when RAILS_SPLIT_ASSETS_BUILD=true, which E2E on CNG sets, so there the rails job does the waiting itself.

  • fetch_assets keeps its early returns (no ASSETS_IMAGE, COMPILE_ASSETS, needs_build false). It then runs wait_for_assets with ASSETS_WAIT_TIMEOUT set to ASSETS_IMAGE_WAIT_TIMEOUT_SECONDS (default still 2700). So the rails jobs now also:
    • wait only for a missing manifest,
    • retry transient registry errors with backoff,
    • fail at once on any other error, with skopeo's message and exit status,
    • fail before polling when the repository path is wrong.
  • On a timeout, fetch_assets now exits with 75 instead of 1. Nothing in the rails jobs depends on the exit code, and .job-base's retry: 1 covers both.
  • The last line of wait_for_assets's timeout message no longer says "the gitlab-rails-* jobs start once it passes", because it now runs inside those jobs too.

How was it tested?

  • Three new tests in build-scripts/tests/wait_for_assets.bats cover what fetch_assets adds on top of wait_for_assets: the needs_build skip, the default 2700s timeout, and that ASSETS_IMAGE_WAIT_TIMEOUT_SECONDS overrides the gate's ASSETS_WAIT_TIMEOUT only for the call. The rest of the wait is already covered by that file. The full suite passes: 110 tests.

  • With the real skopeo 1.2.2, curl and jq in distribution_ci_tools:5.8.0, the image the rails jobs run:

    Case Result
    gitlab-assets-ee:master available on the first poll
    missing tag, timeout 0 exit 75 with the full explanation
    no access (403) fails on the first poll with skopeo's message
    skopeo rejects its arguments fails on the first poll with skopeo's message

Related

Edited by Heinrich Lee Yu

Merge request reports