Fix fetch_assets to authenticate skopeo inspect for private registries
What does this MR do?
Fixes the fetch_assets function in build-scripts/build.sh to pass CI_JOB_TOKEN credentials to skopeo inspect when polling for the assets image.
On the security mirror (gitlab-org/security/charts/components/images), the assets image lives at a private registry path (e.g. registry.gitlab.com/gitlab-org/security/gitlab/gitlab-assets-ee:master) that requires authentication. The skopeo inspect call was made without credentials, causing it to fail with:
Error parsing image name "docker://registry.gitlab.com/gitlab-org/security/gitlab/gitlab-assets-ee:master":
reading manifest master in registry.gitlab.com/gitlab-org/security/gitlab/gitlab-assets-ee:
errors: denied: requested access to the resource is denied
unauthorized: authentication requiredSince skopeo does not automatically use Docker's credential store (populated by docker_login in before_script), the inspect call returned empty output and the polling loop ran until the job timed out.
The fix passes --creds "gitlab-ci-token:${CI_JOB_TOKEN}" to skopeo inspect, matching the authentication already performed by docker_login. The 2>/dev/null suppresses error output from failed inspect attempts (e.g. while the image is not yet available), keeping the polling loop clean.
Investigation findings
Further investigation confirmed that both image availability and authentication are contributing factors:
-
The error message is misleading: The GitLab container registry returns an identical
HTTP 401 UNAUTHORIZEDresponse for both "image exists but authentication required" and "image does not exist in a private namespace". Anonymous token requests forgitlab-org/security/*are explicitly denied (HTTP 403), soskopeocannot distinguish between the two cases without credentials. -
The polling loop was structurally broken for private registries: Without
--creds,skopeo inspectreturns empty stdout even when the image is available — the loop could never exit regardless of image availability. This explains the 2-hour job timeouts. -
The failure is intermittent, not constant: Between the 2026-05-12 and 2026-05-20 failing pipelines, there were 8 successful runs. This is because
needs_build()returns false when thegitlab-rails-eeimage already exists — in those casesfetch_assetsskips the polling loop entirely, so the auth bug is never triggered. -
The failure is pre-existing and systemic: At least 3 confirmed instances of the same
skopeoauth failure pattern on the security mirror (2026-05-12, 2026-05-20, 2026-05-22), with the root cause present since the security mirror was set up.
| Scenario | Without --creds |
With --creds |
|---|---|---|
| Image not yet available | Loop runs forever (401 = empty stdout) | Loop waits correctly (MANIFEST_UNKNOWN = empty stdout) |
| Image becomes available | Loop never exits (still 401) | Loop exits correctly (manifest = non-empty stdout) |
Related issues
Closes #2322 (closed)
Checklist
See Definition of done.
For anything in this list which will not be completed, please provide a reason in the MR discussion
Required
- Merge Request Title, and Description are up to date, accurate, and descriptive
- MR targeting the appropriate branch
- MR has a green pipeline on GitLab.com
- When ready for review, MR is labeled "~workflow::ready for review" per the Distribution MR workflow
Expected (please provide an explanation if not completing)
- Test plan indicating conditions for success has been posted and passes
- Documentation created/updated - N/A (internal build script fix)
- Integration tests added to GitLab QA - N/A
- The impact any change in container size has should be evaluated - N/A (no container changes)
- New dependencies are managed with GitLab forked renovatebot - N/A