Unable to get Maven package if GCS object storage is used and if proxy_download is disabled

Summary

Unable to get Maven package if GCS object storage is used and if proxy_download is disabled. sbt and Maven receive 500 error while trying to download a package. Access via curl using private token does not work as well.

The issue occurs after upgrading Gitlab 15.8 version.

This might be somehow related !106242 (merged)

Gitlab runs on GKE cluster using official helm chart, helm chart version 6.8.0.

Steps to reproduce

1. Configure Gitlab storage using GCS and set proxy_download: false
2. Push any package to maven repo on Gitlab instance
3. Try to access the package using curl with private token or using mvn install

Example Project

What is the current bug behavior?

I receive an error: Internal Server Error (500)

What is the expected correct behavior?

200/HTTP, file is downloaded

Relevant logs and/or screenshots

If I request a package using API:

curl --header "PRIVATE-TOKEN: xxxxx" https://gitlab.example.com/api/v4/projects/1/packages/maven/com/exampleorg/examplepackage/1.0.0/examplepackage_1.0.0.pom

{
"component": "gitlab",
  "subcomponent": "api_json",
  "level": "error",
  "time": "2023-01-25T08:57:44.025Z",
  "severity": "INFO",
  "duration_s": 0.10431,
  "db_duration_s": 0.04084,
  "view_duration_s": 0.06347,
  "status": 500,
  "method": "GET",
  "path": "/api/v4/projects/1/packages/maven/com/exampleorg/examplepackage/maven-metadata.xml",
  "params": [],
  "host": "gitlab.example.com",
  "remote_ip": "x.x.x.x, x.x.x.x",
  "ua": "Aether",
  "route": "/api/:version/projects/:id/packages/maven/*path/:file_name",
  "user_id": 61,
  "username": "project_1_bot",
  "token_type": "PersonalAccessToken",
  "token_id": 15,
  "exception.class": "ArgumentError",
  "exception.message": "wrong number of arguments (given 1, expected 2)",
  "exception.backtrace": [
    "lib/api/maven_packages.rb:100:in `head_request_on_aws_file?'",
    "lib/api/helpers.rb:611:in `present_carrierwave_file!'",
    "lib/api/maven_packages.rb:88:in `present_carrierwave_file_with_head_support!'",
    "lib/api/maven_packages.rb:145:in `block in find_and_present_package_file'",
    "lib/api/helpers/packages/dependency_proxy_helpers.rb:26:in `redirect_registry_request'",
    "lib/api/maven_packages.rb:128:in `find_and_present_package_file'",
    "lib/api/maven_packages.rb:275:in `block (2 levels) in <class:MavenPackages>'",
    "ee/lib/gitlab/middleware/ip_restrictor.rb:14:in `block in call'",
    "ee/lib/gitlab/ip_address_state.rb:10:in `with'",
    "ee/lib/gitlab/middleware/ip_restrictor.rb:13:in `call'",
    "lib/api/api_guard.rb:215:in `call'",
    "lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in `call'",
    "lib/gitlab/middleware/memory_report.rb:13:in `call'",
    "lib/gitlab/middleware/speedscope.rb:13:in `call'",
    "lib/gitlab/database/load_balancing/rack_middleware.rb:23:in `call'",
    "lib/gitlab/middleware/rails_queue_duration.rb:33:in `call'",
    "lib/gitlab/metrics/rack_middleware.rb:16:in `block in call'",
    "lib/gitlab/metrics/web_transaction.rb:46:in `run'",
    "lib/gitlab/metrics/rack_middleware.rb:16:in `call'",
    "lib/gitlab/jira/middleware.rb:19:in `call'",
    "lib/gitlab/middleware/go.rb:20:in `call'",
    "lib/gitlab/etag_caching/middleware.rb:21:in `call'",
    "lib/gitlab/middleware/query_analyzer.rb:11:in `block in call'",
    "lib/gitlab/database/query_analyzer.rb:37:in `within'",
    "lib/gitlab/middleware/query_analyzer.rb:11:in `call'",
    "lib/gitlab/middleware/multipart.rb:173:in `call'",
    "lib/gitlab/middleware/read_only/controller.rb:50:in `call'",
    "lib/gitlab/middleware/read_only.rb:18:in `call'",
    "lib/gitlab/middleware/same_site_cookies.rb:27:in `call'",
    "lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'",
    "lib/gitlab/middleware/basic_health_check.rb:25:in `call'",
    "lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'",
    "lib/gitlab/middleware/request_context.rb:21:in `call'",
    "lib/gitlab/middleware/webhook_recursion_detection.rb:15:in `call'",
    "config/initializers/fix_local_cache_middleware.rb:11:in `call'",
    "lib/gitlab/middleware/compressed_json.rb:37:in `call'",
    "lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'",
    "lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call'",
    "lib/gitlab/metrics/requests_rack_middleware.rb:77:in `call'",
    "lib/gitlab/middleware/release_env.rb:13:in `call'"
  ],
  "queue_duration_s": 0.011617,
  "redis_calls": 3,
  "redis_duration_s": 0.013584,
  "redis_read_bytes": 203,
  "redis_write_bytes": 175,
  "redis_cache_calls": 1,
  "redis_cache_duration_s": 0.007158,
  "redis_cache_read_bytes": 203,
  "redis_cache_write_bytes": 67,
  "redis_shared_state_calls": 2,
  "redis_shared_state_duration_s": 0.006426,
  "redis_shared_state_write_bytes": 108,
  "db_count": 13,
  "db_write_count": 1,
  "db_cached_count": 0,
  "db_replica_count": 0,
  "db_primary_count": 13,
  "db_main_count": 13,
  "db_main_replica_count": 0,
  "db_replica_cached_count": 0,
  "db_primary_cached_count": 0,
  "db_main_cached_count": 0,
  "db_main_replica_cached_count": 0,
  "db_replica_wal_count": 0,
  "db_primary_wal_count": 0,
  "db_main_wal_count": 0,
  "db_main_replica_wal_count": 0,
  "db_replica_wal_cached_count": 0,
  "db_primary_wal_cached_count": 0,
  "db_main_wal_cached_count": 0,
  "db_main_replica_wal_cached_count": 0,
  "db_replica_duration_s": 0,
  "db_primary_duration_s": 0.041,
  "db_main_duration_s": 0.041,
  "db_main_replica_duration_s": 0,
  "cpu_s": 0.062172,
  "mem_objects": 16506,
  "mem_bytes": 1788272,
  "mem_mallocs": 7194,
  "mem_total_bytes": 2448512,
  "pid": 44,
  "worker_id": "puma_1",
  "rate_limiting_gates": [],
  "correlation_id": "01GQM2BDPY0CQ059KHWNRE72ZZ",
  "meta.caller_id": "GET /api/:version/projects/:id/packages/maven/*path/:file_name",
  "meta.remote_ip": "34.89.213.169",
  "meta.feature_category": "package_registry",
  "meta.user": "project_1_bot",
  "meta.user_id": 61,
  "meta.client_id": "user/61",
  "request_urgency": "low",
  "target_duration_s": 5
}

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info

System information
System:
Proxy:          GITLAB_GITLAB_PAGES_SERVICE_PORT_HTTP_GITLAB_PAGES_PROXY: 8090
Current User:   git
Using RVM:      no
Ruby Version:   2.7.7p221
Gem Version:    3.2.33
Bundler Version:2.3.15
Rake Version:   13.0.6
Redis Version:  unknown
Sidekiq Version:6.5.7
Go Version:     unknown

GitLab information
Version:        15.8.0-ee
Revision:       1d89c23c9f9
Directory:      /srv/gitlab
DB Adapter:     PostgreSQL
DB Version:     13.7
URL:            https://gitlab.example.com
HTTP Clone URL: https://gitlab.example.com/some-group/some-project.git
SSH Clone URL:  git@gitlab.example.com:some-group/some-project.git
Elasticsearch:  no
Geo:            no
Using LDAP:     no
Using Omniauth: yes
Omniauth Providers: google_oauth2

GitLab Shell
Version:        14.15.0
Repository storages:
- default:      tcp://gitlab-gitaly-0.gitlab-gitaly.gitlab.svc:8075
GitLab Shell path:              /home/git/gitlab-shell

Results of GitLab application Check

Expand for output related to the GitLab application check

We use Kubernetes and Gitlab helm chart! I guess this is the reason of some failed checks
Checking GitLab subtasks ...
Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 14.15.0 ? ... OK (14.15.0)
Running /home/git/gitlab-shell/bin/check
gitlab-shell self-check failed
Try fixing it:
Make sure GitLab is running;
Check the gitlab-shell configuration file:
sudo -u git -H editor /home/git/gitlab-shell/config.yml
Please fix the error above and rerun the checks.
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes
Number of Sidekiq processes (cluster/worker) ... 1/1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab App ...
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Cable config exists? ... yes
Resque config exists? ... yes
Log directory writable? ... yes
Tmp directory writable? ... no
Try fixing it:
sudo chown -R gitlab /srv/gitlab/tmp
sudo chmod -R u+rwX /srv/gitlab/tmp
For more information see:
doc/install/installation.md in section "GitLab"
Please fix the error above and rerun the checks.
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... skipped (no tmp uploads folder yet)
Systemd unit files or init script exist? ... no
Try fixing it:
Install the Service
For more information see:
doc/install/installation.md in section "Install the Service"
Please fix the error above and rerun the checks.
Systemd unit files or init script up-to-date? ... can't check because of previous errors
Projects have namespace: ...
2/1 ... yes
7/3 ... yes
13/7 ... yes
12/9 ... yes
10/10 ... yes
9/11 ... yes
11/12 ... yes
17/15 ... yes
18/16 ... yes
20/19 ... yes
10/20 ... yes
11/21 ... yes
11/22 ... yes
20/23 ... yes
10/24 ... yes
11/26 ... yes
11/27 ... yes
34/28 ... yes
34/29 ... yes
10/30 ... yes
3/31 ... yes
42/32 ... yes
46/35 ... yes
48/36 ... yes
21/37 ... yes
20/38 ... yes
28/39 ... yes
58/42 ... yes
58/43 ... yes
59/46 ... yes
60/47 ... yes
59/48 ... yes
3/49 ... yes
91/50 ... yes
63/51 ... yes
62/52 ... yes
65/53 ... yes
65/54 ... yes
65/55 ... yes
20/57 ... yes
20/58 ... yes
20/59 ... yes
20/60 ... yes
20/61 ... yes
11/63 ... yes
44/65 ... yes
71/66 ... yes
11/68 ... yes
76/70 ... yes
20/71 ... yes
82/72 ... yes
82/73 ... yes
82/76 ... yes
91/77 ... yes
10/78 ... yes
11/79 ... yes
62/80 ... yes
21/81 ... yes
11/83 ... yes
96/84 ... yes
76/85 ... yes
20/86 ... yes
96/87 ... yes
20/88 ... yes
62/89 ... yes
62/91 ... yes
62/92 ... yes
23/95 ... yes
113/96 ... yes
20/97 ... yes
20/98 ... yes
10/100 ... yes
121/102 ... yes
121/105 ... yes
20/107 ... yes
20/108 ... yes
134/109 ... yes
43/110 ... yes
20/111 ... yes
114/112 ... yes
135/113 ... yes
114/114 ... yes
113/115 ... yes
139/116 ... yes
20/117 ... yes
139/118 ... yes
139/119 ... yes
11/120 ... yes
134/121 ... yes
43/122 ... yes
144/123 ... yes
20/124 ... yes
20/125 ... yes
11/126 ... yes
91/127 ... yes
11/128 ... yes
144/129 ... yes
20/131 ... yes
135/132 ... yes
20/133 ... yes
91/134 ... yes
127/135 ... yes
127/136 ... yes
20/137 ... yes
11/138 ... yes
91/139 ... yes
155/140 ... yes
155/142 ... yes
20/144 ... yes
51/145 ... yes
161/146 ... yes
161/147 ... yes
160/148 ... yes
34/149 ... yes
20/150 ... yes
98/151 ... yes
20/152 ... yes
113/153 ... yes
114/154 ... yes
10/155 ... yes
11/156 ... yes
150/157 ... yes
155/158 ... yes
20/159 ... yes
20/160 ... yes
150/161 ... yes
20/162 ... yes
71/163 ... yes
110/164 ... yes
110/165 ... yes
155/166 ... yes
113/167 ... yes
155/168 ... yes
155/171 ... yes
145/173 ... yes
155/174 ... yes
155/175 ... yes
11/176 ... yes
11/177 ... yes
135/178 ... yes
11/179 ... yes
93/180 ... yes
165/183 ... yes
11/184 ... yes
10/185 ... yes
10/186 ... yes
62/187 ... yes
11/188 ... yes
11/189 ... yes
20/190 ... yes
155/191 ... yes
20/192 ... yes
20/193 ... yes
139/194 ... yes
20/195 ... yes
10/196 ... yes
139/197 ... yes
87/198 ... yes
51/199 ... yes
348/200 ... yes
20/201 ... yes
11/202 ... yes
11/203 ... yes
48/204 ... yes
10/205 ... yes
114/206 ... yes
154/207 ... yes
145/208 ... yes
20/209 ... yes
11/210 ... yes
91/211 ... yes
114/212 ... yes
11/213 ... yes
114/214 ... yes
348/215 ... yes
20/216 ... yes
349/218 ... yes
110/219 ... yes
20/220 ... yes
430/221 ... yes
370/222 ... yes
20/223 ... yes
11/224 ... yes
3/225 ... yes
450/226 ... yes
450/227 ... yes
137/228 ... yes
20/229 ... yes
20/230 ... yes
20/231 ... yes
20/232 ... yes
20/233 ... yes
43/234 ... yes
10/235 ... yes
36/236 ... yes
123/237 ... yes
123/238 ... yes
467/239 ... yes
20/240 ... yes
20/241 ... yes
349/242 ... yes
20/243 ... yes
125/244 ... yes
114/245 ... yes
20/246 ... yes
10/247 ... yes
165/248 ... yes
3/249 ... yes
349/250 ... yes
20/251 ... yes
139/252 ... yes
20/253 ... yes
20/254 ... yes
20/255 ... yes
20/256 ... yes
20/257 ... yes
20/258 ... yes
348/259 ... yes
348/260 ... yes
455/261 ... yes
139/263 ... yes
43/264 ... yes
43/265 ... yes
11/267 ... yes
11/268 ... yes
523/271 ... yes
20/272 ... yes
113/273 ... yes
523/274 ... yes
3/275 ... yes
3/276 ... yes
20/277 ... yes
20/278 ... yes
10/279 ... yes
348/280 ... yes
82/281 ... yes
91/282 ... yes
11/283 ... yes
113/284 ... yes
349/285 ... yes
114/286 ... yes
11/288 ... yes
139/289 ... yes
20/290 ... yes
3/291 ... yes
20/292 ... yes
123/293 ... yes
10/295 ... yes
20/296 ... yes
10/297 ... yes
11/298 ... yes
20/299 ... yes
372/300 ... yes
569/301 ... yes
349/302 ... yes
114/303 ... yes
20/304 ... yes
10/306 ... yes
372/307 ... yes
91/308 ... yes
Redis version >= 6.0.0? ... yes
Ruby version >= 2.7.2 ? ... yes (2.7.7)
Git user has default SSH configuration? ... yes
Active users: ... 182
Is authorized keys file accessible? ... skipped (authorized keys not enabled)
GitLab configured to store new projects in hashed storage? ... yes
All projects are in hashed storage? ... yes
Elasticsearch version 7.x-8.x or OpenSearch version 1.x ... skipped (Advanced Search is disabled)
All migrations must be finished before doing a major upgrade ... skipped (Advanced Search is disabled)
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished

Possible fixes