`CI_JOB_TOKEN` for `npm` does not work with dependencies

Summary

Using a CI_JOB_TOKEN for installation of npm packages with dependencies in another project, will yield 404 errors.
Using Personal Access Tokens works as expected

To install an npm package from a private project, you can authenticate using the CI_JOB_TOKEN, like so:

@foo:registry=https://gitlab.example.com/api/v4/packages/npm/
//gitlab.example.com/api/v4/packages/npm/:_authToken=${CI_JOB_TOKEN}

This authentication is happening through the Instance-level npm endpoint.

Now, if a package has dependencies for packages, in another project, the pipeline will fail with the following error:

npm ERR! code E404
npm ERR! 404 Not Found - GET <XXX>
npm ERR! 404
npm ERR! 404 '<XXX>' is not in this registry.
npm ERR! 404 This package name is not valid, because
npm ERR! 404 1. name can only contain URL-friendly characters
npm ERR! 404
npm ERR! 404 Note that you can also install from a
npm ERR! 404 tarball, folder, http url, or git url.

This is resolved by adding the authentication for the dependent package(s)' project at the Project-level npm endpoint:

//gitlab.example.com/api/v4/projects/${CI_PROJECT_ID}/packages/npm/:_authToken=${CI_JOB_TOKEN}

A customer reaching out to GitLab Support has reported this issue. See the ticket here.

Steps to reproduce

Example Project

What is the current bug behavior?

Dependent packages, in another project, are resolved with 404.

What is the expected correct behavior?

Dependent packages, in another project, are successfully found

Relevant logs and/or screenshots

Output of checks

This bug happens on GitLab.com

Results of GitLab environment info

Expand for output related to GitLab environment info


(For installations with omnibus-gitlab package run and paste the output of:
`sudo gitlab-rake gitlab:env:info`)

(For installations from source run and paste the output of:
`sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production`)

Results of GitLab application Check

Expand for output related to the GitLab application check

(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true)
(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true)
(we will only investigate if the tests are passing)