Skip to content

Dependency proxy error reporting confusing - "404/not found" should be "access denied" when not group member

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

Summary

A customer (ZD internal link) was experiencing 404 errors on builds trying to pull images via the dependency proxy, seemingly at random. This initially led us to suspect there was a problem finding the corresponding files on S3. After considerable investigation, the root cause was traced to some of the users running the builds not being members of the top level group containing the dependency proxy. These users were able to log into the proxy, but not pull the images from it.

While is is documented that users must have at least guest access to the proxy group, the root cause would have been identified more quickly if the build job reported an Access Denied error when trying to pull the image, rather than a 404.

It is noted that when a user tries to pull an image from a project container registry that they do not have access to, they receive a "denied: requested access to the resource is denied" message, not a "manifest unknown" error.

Steps to reproduce

From command line:

  1. A user has developer access to a project to run builds, but is not a member of the parent group (group99) which has dependency proxy enabled.
  2. From command line, user logs into dependency proxy via docker login gitlab.example.com and successfully authenticates.
  3. User then runs docker pull gitlab.example.com/group99/alpine:latest.
  4. Result is Error response from daemon: error parsing HTTP 404 response body: unexpected end of JSON input: "".

From build job:

When job run by above user job log outputs the following message when trying to access alpine:latest image via proxy:

#5 [internal] load metadata for gitlab.example.com:443/group99/dependency_proxy/containers/alpine:latest
#5 sha256:b2d14c7fa30b9b9a59ec4e6339d4215af7da5c75a914963eb0395ad8c158d979
#5 ERROR: gitlab.example.com:443/group99/dependency_proxy/containers/alpine:latest: not found

failed to solve with frontend dockerfile.v0: failed to create LLB definition: gitlab.example.com/group99/dependency_proxy/containers/alpine:latest: not found

Example Project

What is the current bug behavior?

Error message returned indicating image doesn't exist.

What is the expected correct behavior?

Error message returned indicating access to image has been denied.

Relevant logs and/or screenshots

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info 

(For installations with omnibus-gitlab package run and paste the output of:
`sudo gitlab-rake gitlab:env:info`)

(For installations from source run and paste the output of:
`sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production`)

Results of GitLab application Check

Expand for output related to the GitLab application check 
(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true)


(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true)


(we will only investigate if the tests are passing)

Possible fixes

Edited by 🤖 GitLab Bot 🤖