Investigate: Maven repository not working with AWS as object storage and `proxy_download=false`
Summary
A self-managed customer has reported (in ZD internal link) that after upgrading from 15.3.2 to 15.4.3 they started experiencing CI/CD job errors when attempting to pull Maven packages from their GitLab package registry using Gradle 7.3.3.
The error symptoms closely match those described in !27612 (merged) which was deployed in 12.10.
I found !85299 (merged) which was deployed in 15.4 and which involves changes to the same areas of the code modified in the earlier MR, and I wonder if a regression has occurred?
The customer has been able to work around the problem by setting gitlab_rails['object_store']['proxy_download'] = true
but this is not a long-term solution for them due to the increased load this places on the GitLab instance and the increased network traffic between the GitLab instance and AWS.
I have reproduced the issue using curl -L --head
:
In 15.3.2 - 200 OK
root@ip-172-31-18-196:~# curl -L --head --header "PRIVATE-TOKEN: <token>" http://172.31.18.196//api/v4/projects/3/packages/maven/edu/wisc/mvn-2/mvn-2/1.0-SNAPSHOT/maven-metadata.xml
HTTP/1.1 303 See Other
Server: nginx
Date: Sun, 06 Nov 2022 22:19:54 GMT
Content-Type: text/plain
Connection: keep-alive
Cache-Control: no-cache
Location: https://mybucket.s3.ap-southeast-2.amazonaws.com/4e/07/4e07408562bedb8b60ce05c1decfe3ad16b72230967de01f640b7e4729b49fce/packages/3/files/11/maven-metadata.xml?X-Amz-Expires=600&X-Amz-Date=20221106T221954Z&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AWS_ACCESS_KEY_ID%2F20221106%2Fap-southeast-2%2Fs3%2Faws4_request&X-Amz-SignedHeaders=host&X-Amz-Signature=563e7dd7409bb24db2f97e9e13d327ed1ff38a66f8d408c9aedb42c542cf6e8b
Vary: Origin
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Request-Id: 01GH7GEQSP4ZHWS1AMWNQFPTN8
X-Runtime: 0.477666
Strict-Transport-Security: max-age=63072000
Referrer-Policy: strict-origin-when-cross-origin
HTTP/1.1 200 OK
x-amz-id-2: XwMLJ73Ap6FvFprEtQ7jDzxsNnJ6QrtCo5YDyudTfXiUjkeaNx2ixPeelTiSCqqrTJBymf6M90U=
x-amz-request-id: TQX2NXH9F5XDA786
Date: Sun, 06 Nov 2022 22:19:55 GMT
Last-Modified: Thu, 03 Nov 2022 05:10:09 GMT
ETag: "7874ed4bbf955267c2d4359ff2db96a6"
Accept-Ranges: bytes
Content-Type: binary/octet-stream
Server: AmazonS3
Content-Length: 763
In 15.4.3 - 403 Error:
# curl -L --head --header "PRIVATE-TOKEN: <token>" http://172.31.17.200/api/v4/projects/2/packages/maven/edu/wisc/mvn-2/mvn-2/1.0-SNAPSHOT/maven-metadata.xml
HTTP/1.1 302 Found
Server: nginx
Date: Sun, 06 Nov 2022 22:17:31 GMT
Content-Type: text/plain
Content-Length: 492
Connection: keep-alive
Cache-Control: no-cache
Location: https://mybucket.s3.ap-southeast-2.amazonaws.com/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35/packages/3/files/7/maven-metadata.xml?X-Amz-Expires=600&X-Amz-Date=20221106T221731Z&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AWS_ACCESS_KEY_ID%2F20221106%2Fap-southeast-2%2Fs3%2Faws4_request&X-Amz-SignedHeaders=host&X-Amz-Signature=6d79496a017b592487f518d192d0ed9682b528a22b7066ec46af7475479c59a6
Vary: Origin
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Request-Id: 01GH7GAC751NK1KNHYDEQPMD2Z
X-Runtime: 0.598090
Strict-Transport-Security: max-age=63072000
Referrer-Policy: strict-origin-when-cross-origin
HTTP/1.1 403 Forbidden
x-amz-request-id: 9481YMWE75W4EWQ3
x-amz-id-2: 74tmuh1z6CDf+CA/5IglmKSiUa05clS44UThHbh+v5o4Kxo5BoHtxHvEvFDj+iEcrUd+AXHvBz0=
Content-Type: application/xml
Date: Sun, 06 Nov 2022 22:17:31 GMT
Server: AmazonS3
I have observed that in version 15.3.2 the redirect response to a HEAD
request has a status of 303
, while in version 15.4.3 the redirect has a status of 302
. The status returned to a GET request is 302
in both versions.
Steps to reproduce
Example Project
What is the current bug behavior?
Gradle job fails with Could not HEAD 'https://<aws-s3-signed-url>'. Received status code 403 form server: Forbidden
error.
What is the expected correct behavior?
Package should be downloaded from AWS S3 using signed URL.
Relevant logs and/or screenshots
Output of checks
Results of GitLab environment info
Expand for output related to GitLab environment info
(For installations with omnibus-gitlab package run and paste the output of: `sudo gitlab-rake gitlab:env:info`) (For installations from source run and paste the output of: `sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production`)
Results of GitLab application Check
Expand for output related to the GitLab application check
(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true
)(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true
)(we will only investigate if the tests are passing)