Skip to content

Log object storage forbidden errors during artifact downloads

Stan Hu requested to merge sh-log-object-storage-download-forbidden into main

When artifact downloads are redirected to an object storage provider, the runner can encounter a 403 Forbidden error. It can be difficult to understand whether this 403 error was due to a permissions problem on GitLab or an issue with object storage.

We had one customer that had enabled Google's VPC Service Control, and used it to blocked downloads from other buckets. However, it took us a while to figure out the 403 was caused by this feature.

When object storage responds with an error to a GET request, most providers (Amazon, Google, and Azure) return a standard XML message that contains both a code and a message. We now attempt to decode a JSON or XML response when a CI artifact download encounters a 403 error and display the message in the output.

ZD: https://gitlab.zendesk.com/agent/tickets/269580

Sample errors

time="2022-04-12T09:12:47-07:00" level=info msg="Downloading artifacts from coordinator... ok" id=10 responseStatus="200 OK" token=token
2022/04/12 09:12:47 http: superfluous response.WriteHeader call from gitlab.com/gitlab-org/gitlab-runner/network.checkTestArtifactsDownloadHandlerContent (gitlab_test.go:1628)
time="2022-04-12T09:12:47-07:00" level=info msg="Downloading artifacts from coordinator... ok" id=10 responseStatus="200 OK" token=token
time="2022-04-12T09:12:47-07:00" level=error msg="Downloading artifacts from coordinator... forbidden" id=10 responseStatus="403 Forbidden" status="403 Forbidden (SecurityPolicyViolated: Request violates VPC Service Controls)" token=object-s
time="2022-04-12T09:12:47-07:00" level=error msg="Downloading artifacts from coordinator... forbidden" id=10 responseStatus="403 Forbidden" status="403 Forbidden" token=invalid-
time="2022-04-12T09:12:47-07:00" level=error msg="Downloading artifacts from coordinator... not found" id=11 responseStatus="404 Not Found" token=token
time="2022-04-12T09:12:47-07:00" level=error msg="Downloading artifacts from coordinator... forbidden" id=10 responseStatus="403 Forbidden" status="GET http://127.0.0.1:54820/api/v4/jobs/10/artifacts: 403 Forbidden (not allowed)" token=object-s
time="2022-04-12T09:12:47-07:00" level=error msg="Downloading artifacts from coordinator... forbidden" id=10 responseStatus="403 Forbidden" status="403 Forbidden (application/xml decode error: EOF)" token=object-s
time="2022-04-12T09:12:47-07:00" level=error msg="Downloading artifacts from coordinator... forbidden" id=10 responseStatus="403 Forbidden" status="403 Forbidden (text/xml decode error: expected element type <Error> but have <Test>)" token=object-s
Edited by Stan Hu

Merge request reports