CI_JOB_TOKEN cannot be used to download images from other projects

We are having a scheduled pipeline that went OK until upgrading our on-prem self-hosted Gitlab-ee to 17.10.3.

The pipeline would pull an image belonging to and generated by an other project and use it as the image in our docker runner to run the job.

Of course, the Job token permissions were and are set up correctly. Audit logs shows that the last successful access or use of the job tokens was right before upgrading to 17.10.3.

What is the current bug behavior?

Instead of pulling the image to use it for job execution we are receiving the following error message:

Pulling docker image [image name] ...
WARNING: Failed to pull image with policy "if-not-present": Error response from daemon: Head "[image manifest URL]": unauthorized: HTTP Basic: Access denied. If a password was provided for Git authentication, the password was incorrect or you're required to use a token instead of a password. If a token was provided, it was either incorrect, expired, or improperly scoped. See https://git.hackate.com/help/user/profile/account/two_factor_authentication_troubleshooting.md#error-http-basic-access-denied-if-a-password-was-provided-for-git-authentication- (manager.go:254:0s)
ERROR: Job failed: failed to pull image "[image name]" with specified policies [if-not-present]: Error response from daemon: Head "[image manifest URL]": unauthorized: HTTP Basic: Access denied. If a password was provided for Git authentication, the password was incorrect or you're required to use a token instead of a password. If a token was provided, it was either incorrect, expired, or improperly scoped. See https://git.hackate.com/help/user/profile/account/two_factor_authentication_troubleshooting.md#error-http-basic-access-denied-if-a-password-was-provided-for-git-authentication- (manager.go:254:0s)

What is the expected correct behavior?

Downloading the image, using the CI_JOB_TOKEN how it did before.

Relevant logs and/or screenshots

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info

System information
System:         Ubuntu 22.04
Proxy:          no
Current User:   git
Using RVM:      no
Ruby Version:   3.2.5
Gem Version:    3.6.5
Bundler Version:2.6.5
Rake Version:   13.0.6
Redis Version:  7.0.15
Sidekiq Version:7.2.4
Go Version:     unknown

GitLab information
Version:        17.10.3-ee
Revision:       ed705b29e5c
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     PostgreSQL
DB Version:     14.17
URL:            https://git.hackate.com
HTTP Clone URL: https://git.hackate.com/some-group/some-project.git
SSH Clone URL:  git@git.hackate.com:some-group/some-project.git
Elasticsearch:  no
Geo:            no
Using LDAP:     no
Using Omniauth: yes
Omniauth Providers:

GitLab Shell
Version:        14.41.0
Repository storages:
- default:      unix:/var/opt/gitlab/gitaly/gitaly.socket
GitLab Shell path:              /opt/gitlab/embedded/service/gitlab-shell

Gitaly
- default Address:      unix:/var/opt/gitlab/gitaly/gitaly.socket
- default Version:      17.10.3
- default Git Version:  2.48.1.gl1

Results of GitLab application Check

Expand for output related to the GitLab application check

Checking GitLab subtasks ...
Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 14.41.0 ? ... OK (14.41.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/gitlab-shell-check
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes
Number of Sidekiq processes (cluster/worker) ... 1/1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab App ...
Database config exists? ... yes
Tables are truncated? ... skipped
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Cable config exists? ... yes
Resque config exists? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Systemd unit files or init script exist? ... skipped (omnibus-gitlab has neither init script nor systemd units)
Systemd unit files or init script up-to-date? ... skipped (omnibus-gitlab has neither init script nor systemd units)
Projects have namespace: ...
25/16 ... yes
25/17 ... yes
25/18 ... yes
25/19 ... yes
25/20 ... yes
25/21 ... yes
25/22 ... yes
25/23 ... yes
25/24 ... yes
25/25 ... yes
25/26 ... yes
25/27 ... yes
51/28 ... yes
53/29 ... yes
5/30 ... yes
69/32 ... yes
69/33 ... yes
51/34 ... yes
74/35 ... yes
51/36 ... yes
25/37 ... yes
53/38 ... yes
25/40 ... yes
5/41 ... yes
51/42 ... yes
11/43 ... yes
69/44 ... yes
25/45 ... yes
69/47 ... yes
5/48 ... yes
25/49 ... yes
25/50 ... yes
51/52 ... yes
11/54 ... yes
18/55 ... yes
11/57 ... yes
25/58 ... yes
127/59 ... yes
7/60 ... yes
69/61 ... yes
22/63 ... yes
25/64 ... yes
25/66 ... yes
25/67 ... yes
25/68 ... yes
69/69 ... yes
69/70 ... yes
69/71 ... yes
7/72 ... yes
25/73 ... yes
Redis version >= 6.2.14? ... yes
Ruby version >= 3.0.6 ? ... yes (3.2.5)
Git user has default SSH configuration? ... yes
Active users: ... 32
Is authorized keys file accessible? ... yes
GitLab configured to store new projects in hashed storage? ... yes
All projects are in hashed storage? ... yes
Elasticsearch version 7.x-8.x or OpenSearch version 1.x ... skipped (Advanced Search is disabled)
All migrations must be finished before doing a major upgrade ... skipped (Advanced Search is disabled)
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished

Possible fixes