Cannot download generic package using job token provided via Basic auth

Summary

In a CI pipeline, we needed to download a package from a generic package registry as part of our build process. We were using CMake, which was configured to use a .netrc file to look up authentication credentials, which causes CMake to use Basic auth to download the file with those credentials. This worked fine on developer's systems when using a personal access token, but failed in the CI environment when using a job token.

Steps to reproduce

1. Create an internal project
2. Upload a file to the generic package registry for that project
3. Create a CI job which downloads that file using Basic auth with the job token as password
4. Run the job, and note that it fails
5. Change the job to provide the job token via the JOB-TOKEN header
6. Note that the job can now successfully download the file

Example Project

I have created https://gitlab.com/brcampbell1/auth-test-project to demonstrate the issue; see this job https://gitlab.com/brcampbell1/auth-test-project/-/jobs/2935965118 from this .gitlab-ci.yml: https://gitlab.com/brcampbell1/auth-test-project/-/blob/main/.gitlab-ci.yml

Note that it was a private project, not an internal project, as was the case on our local instance; because I can't make a project internal on GitLab.com. Also note that I have made it public after reproducing the problem so that I can link to it from here.

I tried to use this project to demonstrate an alternate path that worked; on our internal system, both auth methods worked for PyPI packages, while only JOB-TOKEN worked for generic pacakge registry. But for some reason on this private project, neither auth method works for PyPI packages, though we still see that JOB-TOKEN works for the generic package registry. I don't know why PyPI fails on GitLab.com but works on our hosted site; it's possible that it's a difference between private and internal projects, but I haven't dug into that, that's a different issue than what I'm reporting here.

What is the current bug behavior?

I cannot download a package from the generic package registry using basic auth and the job token.

What is the expected correct behavior?

I would expect to be able to download packages from the generic package registry using basic auth and the job token, since it works using basic auth and a personal access token, and at least on our self-hosted instance, it works with basic auth and a job token for our PyPI package registry.

In general, I would expect basic auth to be usable for all kinds of tokens consistently; while I was able to work around this issue by configuring CMake to use the header to authenticate, there are some tools that may only support .netrc/basic auth, and it would have saved a lot of time debugging this issue if basic auth were supported.

Relevant logs and/or screenshots

Using docker image sha256:9979aa7673a8676cdef4a0a9771d997d4a365a91c37f25671eef11386f6ef3ab for curlimages/curl:latest with digest curlimages/curl@sha256:5a2a25d96aa941ea2fc47acc50122f7c3d007399a075df61a82d6d2c3a567a2b ...
+ echo '$ $CURL_COMMAND --header "JOB-TOKEN: $CI_JOB_TOKEN" $PYPI_PACKAGE_URL || true'
+ curl --no-progress-meter --fail-with-body -o /dev/null --header 'JOB-TOKEN: [MASKED]' https://gitlab.com/api/v4/projects/38884561/packages/pypi/files/49e6fec125cf8341d305e9923caaf687ca59d0d0f6ed808dd58b1e643cbe2ab5/auth-test-project-0.1.0.tar.gz
$ set -x
$ $CURL_COMMAND --header "JOB-TOKEN: $CI_JOB_TOKEN" $PYPI_PACKAGE_URL || true
curl: (22) The requested URL returned error: 401
+ true
+ echo '$ $CURL_COMMAND -u "ci-job-token:$CI_JOB_TOKEN" $PYPI_PACKAGE_URL || true'
$ $CURL_COMMAND -u "ci-job-token:$CI_JOB_TOKEN" $PYPI_PACKAGE_URL || true
+ curl --no-progress-meter --fail-with-body -o /dev/null -u ci-job-token:[MASKED] https://gitlab.com/api/v4/projects/38884561/packages/pypi/files/49e6fec125cf8341d305e9923caaf687ca59d0d0f6ed808dd58b1e643cbe2ab5/auth-test-project-0.1.0.tar.gz
curl: (22) The requested URL returned error: 401
+ true
+ echo '$ $CURL_COMMAND --header "JOB-TOKEN: $CI_JOB_TOKEN" $GENERIC_PACKAGE_URL || true'
$ $CURL_COMMAND --header "JOB-TOKEN: $CI_JOB_TOKEN" $GENERIC_PACKAGE_URL || true
+ curl --no-progress-meter --fail-with-body -o /dev/null --header 'JOB-TOKEN: [MASKED]' https://gitlab.com/api/v4/projects/38884561/packages/generic/auth-test-project/0.1.0/auth-test-project-0.1.0.tar.gz
+ echo '$ $CURL_COMMAND -u "ci-job-token:$CI_JOB_TOKEN" $GENERIC_PACKAGE_URL || true'
$ $CURL_COMMAND -u "ci-job-token:$CI_JOB_TOKEN" $GENERIC_PACKAGE_URL || true
+ curl --no-progress-meter --fail-with-body -o /dev/null -u ci-job-token:[MASKED] https://gitlab.com/api/v4/projects/38884561/packages/generic/auth-test-project/0.1.0/auth-test-project-0.1.0.tar.gz
curl: (22) The requested URL returned error: 401
+ true

Output of checks

This bug happens on GitLab.com

Results of GitLab environment info

Expand for output related to GitLab environment info

System information
System:		Ubuntu 20.04
Proxy:		no
Current User:	git
Using RVM:	no
Ruby Version:	2.7.5p203
Gem Version:	3.1.6
Bundler Version:2.3.15
Rake Version:	13.0.6
Redis Version:	6.2.7
Sidekiq Version:6.4.0
Go Version:	unknown

GitLab information
Version:	15.3.1-ee
Revision:	518311979e3
Directory:	/opt/gitlab/embedded/service/gitlab-rails
DB Adapter:	PostgreSQL
DB Version:	12.8
URL:		https://gitlab.beta.team
HTTP Clone URL:	https://gitlab.beta.team/some-group/some-project.git
SSH Clone URL:	git@gitlab.beta.team:some-group/some-project.git
Elasticsearch:	yes
Geo:		no
Using LDAP:	no
Using Omniauth:	yes
Omniauth Providers: azure_oauth2

GitLab Shell
Version:	14.10.0
Repository storage paths:
- default: 	/var/opt/gitlab/git-data/repositories
GitLab Shell path:		/opt/gitlab/embedded/service/gitlab-shell

Results of GitLab application Check

Expand for output related to the GitLab application check

ubuntu@ip-10-250-85-6:~$ sudo gitlab-rake gitlab:check SANITIZE=true
Checking GitLab subtasks ...
Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 14.10.0 ? ... OK (14.10.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes
Number of Sidekiq processes (cluster/worker) ... 1/1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab App ...
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Systemd unit files or init script exist? ... skipped (omnibus-gitlab has neither init script nor systemd units)
Systemd unit files or init script up-to-date? ... skipped (omnibus-gitlab has neither init script nor systemd units)
Projects have namespace: ...
2/5 ... yes
10/6 ... yes
8/7 ... yes
8/13 ... yes
25/14 ... yes
3/15 ... yes
25/16 ... yes
3/17 ... yes
27/18 ... yes
27/19 ... yes
27/20 ... yes
27/21 ... yes
27/22 ... yes
36/24 ... yes
36/25 ... yes
35/26 ... yes
35/27 ... yes
35/28 ... yes
25/29 ... yes
3/30 ... yes
37/31 ... yes
25/32 ... yes
25/33 ... yes
35/34 ... yes
25/35 ... yes
37/38 ... yes
37/39 ... yes
37/40 ... yes
37/41 ... yes
37/42 ... yes
37/43 ... yes
37/44 ... yes
37/46 ... yes
37/48 ... yes
10/49 ... yes
4/51 ... yes
169/52 ... yes
25/54 ... yes
10/55 ... yes
169/57 ... yes
169/58 ... yes
169/59 ... yes
3/60 ... yes
26/61 ... yes
25/63 ... yes
3/66 ... yes
25/67 ... yes
26/68 ... yes
25/69 ... yes
25/70 ... yes
25/72 ... yes
25/73 ... yes
4/74 ... yes
25/75 ... yes
38/76 ... yes
48/77 ... yes
25/78 ... yes
25/79 ... yes
25/80 ... yes
25/81 ... yes
25/82 ... yes
25/83 ... yes
25/84 ... yes
34/86 ... yes
4/88 ... yes
27/89 ... yes
27/90 ... yes
27/91 ... yes
27/93 ... yes
81/94 ... yes
25/95 ... yes
27/96 ... yes
25/97 ... yes
79/99 ... yes
38/100 ... yes
25/101 ... yes
25/102 ... yes
25/133 ... yes
25/134 ... yes
25/135 ... yes
25/136 ... yes
48/137 ... yes
25/138 ... yes
65/139 ... yes
76/140 ... yes
26/141 ... yes
107/142 ... yes
107/143 ... yes
76/144 ... yes
10/146 ... yes
25/147 ... yes
3/148 ... yes
76/149 ... yes
25/150 ... yes
27/151 ... yes
27/152 ... yes
64/154 ... yes
66/155 ... yes
27/156 ... yes
37/157 ... yes
25/159 ... yes
81/160 ... yes
5/161 ... yes
25/162 ... yes
35/163 ... yes
10/164 ... yes
26/165 ... yes
37/166 ... yes
26/167 ... yes
81/168 ... yes
25/169 ... yes
25/170 ... yes
9/171 ... yes
27/172 ... yes
27/173 ... yes
3/174 ... yes
25/175 ... yes
25/176 ... yes
25/177 ... yes
9/178 ... yes
3/179 ... yes
140/180 ... yes
10/181 ... yes
25/182 ... yes
10/184 ... yes
25/186 ... yes
140/187 ... yes
140/188 ... yes
140/189 ... yes
140/190 ... yes
140/191 ... yes
27/192 ... yes
10/193 ... yes
237/196 ... yes
25/199 ... yes
50/200 ... yes
10/203 ... yes
25/205 ... yes
25/207 ... yes
25/208 ... yes
25/211 ... yes
169/212 ... yes
25/214 ... yes
10/218 ... yes
10/220 ... yes
10/221 ... yes
10/224 ... yes
81/227 ... yes
27/228 ... yes
27/229 ... yes
3/230 ... yes
25/232 ... yes
25/233 ... yes
25/235 ... yes
81/236 ... yes
25/237 ... yes
158/238 ... yes
25/239 ... yes
25/240 ... yes
10/241 ... yes
25/242 ... yes
25/243 ... yes
25/244 ... yes
25/245 ... yes
37/246 ... yes
10/247 ... yes
37/248 ... yes
37/249 ... yes
25/250 ... yes
25/251 ... yes
25/252 ... yes
25/253 ... yes
25/254 ... yes
51/255 ... yes
35/256 ... yes
10/257 ... yes
25/258 ... yes
26/259 ... yes
27/262 ... yes
648/263 ... yes
10/264 ... yes
27/265 ... yes
27/266 ... yes
26/267 ... yes
25/268 ... yes
162/271 ... yes
162/272 ... yes
164/274 ... yes
25/275 ... yes
25/276 ... yes
25/277 ... yes
19/278 ... yes
25/279 ... yes
50/281 ... yes
50/282 ... yes
10/283 ... yes
25/284 ... yes
147/286 ... yes
26/287 ... yes
26/288 ... yes
162/290 ... yes
162/291 ... yes
27/295 ... yes
27/296 ... yes
27/297 ... yes
181/298 ... yes
147/300 ... yes
147/301 ... yes
147/302 ... yes
164/303 ... yes
25/304 ... yes
25/305 ... yes
25/306 ... yes
27/308 ... yes
155/310 ... yes
25/312 ... yes
101/313 ... yes
8/314 ... yes
205/317 ... yes
205/318 ... yes
205/319 ... yes
169/320 ... yes
122/321 ... yes
219/322 ... yes
205/323 ... yes
162/324 ... yes
171/325 ... yes
162/326 ... yes
25/327 ... yes
24/328 ... yes
3/329 ... yes
27/330 ... yes
237/332 ... yes
25/333 ... yes
25/334 ... yes
27/335 ... yes
147/336 ... yes
162/337 ... yes
25/338 ... yes
169/339 ... yes
25/340 ... yes
25/342 ... yes
169/343 ... yes
25/345 ... yes
530/346 ... yes
162/347 ... yes
529/348 ... yes
131/364 ... yes
570/365 ... yes
27/366 ... yes
101/367 ... yes
25/369 ... yes
162/370 ... yes
25/372 ... yes
64/374 ... yes
25/376 ... yes
25/377 ... yes
25/378 ... yes
155/379 ... yes
3/380 ... yes
35/381 ... yes
37/382 ... yes
162/384 ... yes
162/385 ... yes
162/387 ... yes
616/388 ... yes
101/389 ... yes
615/390 ... yes
262/391 ... yes
109/392 ... yes
636/393 ... yes
570/394 ... yes
609/395 ... yes
171/397 ... yes
169/398 ... yes
621/399 ... yes
147/400 ... yes
147/401 ... yes
147/402 ... yes
147/403 ... yes
147/404 ... yes
660/406 ... yes
81/407 ... yes
171/408 ... yes
19/409 ... yes
205/410 ... yes
162/411 ... yes
27/412 ... yes
147/413 ... yes
169/414 ... yes
147/415 ... yes
615/417 ... yes
624/418 ... yes
262/419 ... yes
624/420 ... yes
3/421 ... yes
25/422 ... yes
25/423 ... yes
81/424 ... yes
766/425 ... yes
25/426 ... yes
25/427 ... yes
621/428 ... yes
25/429 ... yes
25/431 ... yes
140/433 ... yes
9/435 ... yes
35/436 ... yes
613/437 ... yes
25/438 ... yes
25/441 ... yes
766/442 ... yes
219/444 ... yes
9/446 ... yes
35/447 ... yes
35/448 ... yes
25/449 ... yes
140/450 ... yes
162/451 ... yes
759/452 ... yes
766/453 ... yes
162/455 ... yes
3/456 ... yes
3/458 ... yes
613/462 ... yes
33/463 ... yes
162/464 ... yes
25/465 ... yes
81/466 ... yes
27/467 ... yes
169/468 ... yes
766/469 ... yes
25/470 ... yes
25/471 ... yes
101/472 ... yes
250/473 ... yes
162/474 ... yes
37/475 ... yes
763/476 ... yes
812/477 ... yes
621/478 ... yes
621/479 ... yes
570/480 ... yes
759/481 ... yes
138/482 ... yes
Redis version >= 6.0.0? ... yes
Ruby version >= 2.7.2 ? ... yes (2.7.5)
Git user has default SSH configuration? ... yes
Active users: ... 208
Is authorized keys file accessible? ... yes
GitLab configured to store new projects in hashed storage? ... yes
All projects are in hashed storage? ... yes
Elasticsearch version 7.x-8.x or OpenSearch version 1.x ... yes (elasticsearch 7.10.2)
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished

Possible fixes