Gitlab Omnibus: Docker pulls fails quite often with: "... unauthorized to access repository ..." works with click "retry on job
Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.
Hello,
Summary
Since a few versions (maybe starting around 16.x) .. we have quite often the issue, that on our omnibus installation, the jobs which includes Docker, fails quite often with with "... unauthorized to access repository ...". In all cases, it can be solved with just "retry" the failed stage two or three times.
Proposal
Additional details
- Debian Buster
- Gitlab Omnibus gitlab-ce 16.3.2-ce.0
- Gitlab Runner 16.3.0
- Is this EE or CE?
-
EE -
CE
-
Running with gitlab-runner 16.3.0 (8ec04662)
on Shared Docker Runner 01 66kAj_sP, system ID: s_8dd70....
Preparing the "docker" executor
00:02
Using Docker executor with image fra-test-harbor.example.local/testing/levant/release:latest ...
Pulling docker image fra-test-harbor.example.local/testing/levant/release:latest ...
WARNING: Failed to pull image with policy "always": Error response from daemon: unauthorized: unauthorized to access repository: testing/levant/release, action: pull: unauthorized to access repository: testing/levant/release, action: pull (manager.go:237:0s)
ERROR: Job failed: failed to pull image "fra-test-harbor.example.local/testing/levant/release:latest" with specified policies [always]: Error response from daemon: unauthorized: unauthorized to access repository: testing/levant/release, action: pull: unauthorized to access repository: testing/levant/release, action: pull (manager.go:237:0s)First we thought, that we can solve the issue, with moving the required images from Gitlab registry to our external one and add authentication to before_script, but it seems to be happen, before the image gets even started. so also not no debug inside the stage can help. I've found this issue: gitlab-com/gl-infra/production#15955 (closed), and it was marked as solved.
Since, its quite annoying, I thinking to move from docker to podman and check, if that helps .. but .. maybe not. It also doesn't matter, if the runner on the Gitlab host itself is used, or on separated runner VMs.
So, what can we do, to help to find the issue ?