403 error when HEAD API request for generic packages with AWS as object storage and `proxy_download=false`

Summary

In !27612 (merged) a bug was fixed to allow Gradle to issue HEAD requests for Maven packages stored in the package registry on AWS S3 storage, and with proxy_download disabled. The problem was related to the way the redirects generated by GitLab to S3 were signed.

This issue highlights that the same issue exists for generic packages (and possibly other package types) and that the same functionality should be available to support the use of a HEAD request to obtain key metadata values (e.g. size, etag) to assist with detecting when packages are updated in the registry.

Note that when proxy_download is enabled GitLab retrieves the details from S3 and passes them back to the requesting client, and there is no problem. But enabling this setting is not always an option for customers due to the impact it has on server load and network traffic and associated costs.

GET requests to retrieve the package file work OK whether proxy_download is enabled or not.

This issue was reported by a self-managed Premium customer (ZD internal link).

Note that there is currently an open issue relating to HEAD requests for Maven projects no longer working in 15.4.

Steps to reproduce

  1. Configure the GitLab package registry to store package files in an S3 bucket.

  2. Set gitlab_rails['object_store']['proxy_download'] = false in GitLab to disable proxying of requests to object storage. Doing this means a GitLab will return a redirect response to the client directing it to retrieve the package file from S3 directly, with temporary credentials.

  3. Publish package file as a generic package to the GitLab package registry, e.g.:

    curl --header "PRIVATE-TOKEN: <token>" --upload-file packagefile.pkg    
"https://gitlab.example.com/api/v4/projects/24/packages/generic/my_package/0.0.1/packagefile.pkg"

  4. Run the following HEAD command via curl:

    curl  -L --head --header "PRIVATE-TOKEN: <token>" "https://gitlab.example.com/api/v4/projects/24/packages/generic/my_package/0.0.1/pkgfile.pkg

  5. Observe that a redirect to AWS S3 is received from GitLab and the response to that is a 403 error, e.g.:

    HTTP/1.1 302 Found
Server: nginx
Date: Thu, 10 Nov 2022 08:19:21 GMT
Content-Type: text/plain
Content-Length: 518
Connection: keep-alive
Cache-Control: no-cache
Location: https://ubuntu-2004-15-4-3-registry.s3.ap-southeast-2.amazonaws.com/2c/62/2c624232cdd221771294dfbb310aca000a0df6ac8b66b696d90ef06fdefb64a3/packages/24/files/9/pkgfile.pkg?X-Amz-Expires=600&X-Amz-Date=20221110T081921Z&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=XXXXXXXXXXXXXXXXX%2F20221110%2Fap-southeast-2%2Fs3%2Faws4_request&X-Amz-SignedHeaders=host&X-Amz-Signature=7bea0ecc102039ddceba7ab9d49dbb39b2a685e7763d85e4aaa9ad0b4a700111
Vary: Origin
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Request-Id: 01GHG9YGSDPSBG50S2MRED19G0
X-Runtime: 0.084538
Strict-Transport-Security: max-age=63072000
Referrer-Policy: strict-origin-when-cross-origin

HTTP/1.1 403 Forbidden
x-amz-request-id: 2MZW7GXKM1M87KPW
x-amz-id-2: os5Gro+WgqLmqhW4qDVstAmUkZI1y2NWjUZM0wZ73M40DGWA/SGkoEP6RAKJZRCcHuRLl4ySpQg=
Content-Type: application/xml
Date: Thu, 10 Nov 2022 08:09:58 GMT
Server: AmazonS3

Example Project

What is the current bug behavior?

HEAD requests receive a 403 error from AWS S3 when using the redirect link supplied by GitLab.

What is the expected correct behavior?

HEAD requests should return the expected data and a 200 code.

Relevant logs and/or screenshots

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info 

(For installations with omnibus-gitlab package run and paste the output of:
`sudo gitlab-rake gitlab:env:info`)

(For installations from source run and paste the output of:
`sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production`)

Results of GitLab application Check

Expand for output related to the GitLab application check 
(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true)


(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true)


(we will only investigate if the tests are passing)

Possible fixes

Edited by Justin Farmiloe