Skip to content

NPM package requests forwarding

David Fernandez requested to merge 55344-add-npm-package-requests-forwarding into master

What does this MR do?

This MR adds NPM package requests forwarding. This has an impact in requests done by npm and yarn when asking for package metadata. This is done during package installation for packages scopes configured to use the GitLab NPM repository. See https://docs.gitlab.com/ee/user/packages/npm_registry/#installing-a-package.

If enabled, when the backend doesn't find the requested package, it will answer with a redirect to registry.npmjs.org instead of a forbidden response.

This forwarding is controlled by:

  • An application setting: npm_package_requests_forwarding. Added by !25346 (merged).
  • A feature flag: forward_npm_package_registry_requests. This is an additional safeguard to quickly disable this feature if an issue occurs when enabling it on GitLab.com.

This MR adds the following:

  • A helper API::Helpers::Packages::DependencyProxyHelpers with a function #redirect_registry_request that takes some parameters and a block. If the package is not found and other conditions are met, this function will reply a redirect to npmjs.org for the same package name. In the other cases, the block is executed.
  • Update the NPM API route get 'packages/npm/*package_name' to use the above function.
  • Relevant tests.

See the relevant issue: #55344 (closed).

Being behind a feature flag , this MR doesn't have a changelog or documentation.

Design choices:

  • Basically, when we receive a request for an unknown package, we will forward this request to npmjs.org. That's what (more or less) the dependency proxy is doing for container images. That's why the API helpers is called DependencyProxyHelpers.
  • We went with the dumb and simple solution: the backend answers a redirect response.
  • An alternative would be fetch the response from npmjs.org with the Gitlab::HTTP client and transmit it back but this is a much more complex/involved implementation. This could be useful fo future features (see next point)
  • Note that, for now, we're only answering a redirect but in future, we might add more features to this. Inspired by the Dependency Proxy, we could add an intermediary cache that could serve as a safeguard when npmjs.org is down. As such, we will be creating corresponding services. To centralize all this logic, a helper function works well for APIs.
  • For now, only the npm packages are concerned by this change but we could expand this functionality to other package types. That's why, the helpers function (see #redirect_registry_request) is like a wrapper with a block acting as a fallback when the forward/redirect is disabled. This way, it's really trivial to add support of this feature in the other package types.

Screenshots

For the following examples, the situation is as follows:

  • .npmrc file configured like this:
@babel:registry=https://gitlab.local:3443/api/v4/packages/npm/
//gitlab.local:3443/api/v4/packages/npm/:_authToken=${TOKEN}
  • Notice that the registry url for packages in the @babel scope is set to our local GitLab installation.
  • Our local GitLab installation has 0 npm packages:
Packages::Package.npm.count
   (0.9ms)  SELECT COUNT(*) FROM "packages_packages" WHERE "packages_packages"."package_type" = 2
=> 0
  • We will try to install @babel/core with npm and yarn

With npm package forwarding disabled

$ TOKEN="XXXXX" npm install @babel/core
npm ERR! code E403
npm ERR! 403 403 Forbidden - GET https://gitlab.local:3443/api/v4/packages/npm/@babel%2fcore
npm ERR! 403 In most cases, you or one of your dependencies are requesting
npm ERR! 403 a package version that is forbidden by your security policy.
$ TOKEN="XXXX" yarn add @babel/core
yarn add v1.19.0
info No lockfile found.
[1/4] 🔍  Resolving packages...
error An unexpected error occurred: "https://gitlab.local:3443/api/v4/packages/npm/@babel%2fcore: 403 Forbidden".
info If you think this is a bug, please open a bug report with the information provided in "/Users/david/projects/local_lambs/npm/sandbox/yarn-error.log".
info Visit https://yarnpkg.com/en/docs/cli/add for documentation about this command.

With npm package forwarding enabled

$ TOKEN="XXX" npm install @babel/core
npm notice created a lockfile as package-lock.json. You should commit this file.
npm WARN sandbox@1.0.0 No description
npm WARN sandbox@1.0.0 No repository field.

+ @babel/core@7.8.4
added 36 packages from 56 contributors and audited 198 packages in 23.216s

2 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities
$ TOKEN="XXX" yarn add @babel/core
yarn add v1.19.0
info No lockfile found.
[1/4] 🔍  Resolving packages...
[2/4] 🚚  Fetching packages...
[3/4] 🔗  Linking dependencies...
[4/4] 🔨  Building fresh packages...
success Saved lockfile.
success Saved 26 new dependencies.
info Direct dependencies
└─ @babel/core@7.8.4
info All dependencies
├─ @babel/core@7.8.4
├─ @babel/helper-function-name@7.8.3
├─ @babel/helper-get-function-arity@7.8.3
├─ @babel/helper-split-export-declaration@7.8.3
├─ @babel/helpers@7.8.4
├─ @babel/highlight@7.8.3
├─ ansi-styles@3.2.1
├─ chalk@2.4.2
├─ color-convert@1.9.3
├─ color-name@1.1.3
├─ convert-source-map@1.7.0
├─ escape-string-regexp@1.0.5
├─ gensync@1.0.0-beta.1
├─ globals@11.12.0
├─ has-flag@3.0.0
├─ js-tokens@4.0.0
├─ jsesc@2.5.2
├─ json5@2.1.1
├─ minimist@1.2.0
├─ ms@2.1.2
├─ path-parse@1.0.6
├─ resolve@1.15.1
├─ safe-buffer@5.1.2
├─ semver@5.7.1
├─ supports-color@5.5.0
└─ to-fast-properties@2.0.0
✨  Done in 7.27s.

Notice (on the yarn log) that @babel/core will pull other @babel packages. This is a non issue as they will also receive the redirect to npmjs.org and resolve properly.

Does this MR meet the acceptance criteria?

Conformity

Availability and Testing

Security

If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:

  • Label as security and @ mention @gitlab-com/gl-security/appsec
  • The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
  • Security reports checked/validated by a reviewer from the AppSec team

Closes #55344 (closed)

Edited by David Fernandez

Merge request reports