Skip to content

Lift the NPM package naming convention for the project level API

David Fernandez requested to merge 33685-lift-npm-package-naming-convention-for-project-level-api into master

🍰 Context

The NPM package registry is available at two "levels":

  • The project level. Users will point $ npm to https://gitlab.example.com/api/v4/projects/<project_id>/packages/npm/
  • The instance level. Users will point $ npm to https://gitlab.example.com/api/v4/packages/npm/

At the instance level, the url doesn't give any indication about the Group or the Project containing the package. Instead of searching through all available packages, we restrict available packages using a naming convention. Among other things, this helps us to have a hint on where the package is located.

For NPM, the package scope must be the root group path. This restriction on the package name is enforced at the model level. This part of the code being before my time, I'm not sure why it has been implemented as a model validator.

It causes issues for the project level where users are confused as to why they need to conform to the naming convention. They should be able to name their packages as they see fit. That's exactly issue #33685 (closed).

To be complete, here is the plan for this MR:

On master With this MR
Instance level Packages must follow the naming convention.

If the request forward is enabled, requests on non existing packages are forwarded to npmjs.org		 Packages must follow the naming convention.


If the request forward is enabled, requests on non existing packages are forwarded to npmjs.org
Project level Packages must follow the naming convention.


If the request forward is enabled, requests on non existing packages are forwarded to npmjs.org		 Package names can be any name (including unscoped)


If the request forward is enabled, requests on non existing packages are forwarded to npmjs.org

As you can see, at the instance level, the behavior is strictly the same.

🔬 What does this MR do?

  • Remove the model validation enforcing the naming convention.
    • Replace it by a simple name validation.
  • On the instance level API, only return packages that are within the naming convention.
    • We do this by locating the Group and then searching the package name within that group. The package finder will thus be updated to accept a Project or a Namespace. Yes, it's a Namespace because we need to include user namespaces too so that personal projects are searched when looking for an npm package.
  • Update the relevant specs.
    • In particular, prove that packages not following the naming convention are not returned by the instance level API.
    • Note that the npm specs have been heavily refactored in !53491 (merged)
  • Update the relevant documentation.

📸 Screenshots (strongly suggested)

I created locally a project under the root user namespace and I uploaded 3 npm packages:

Screenshot_2021-02-09_at_15.37.42

Project level

Let's try to pull those 3 packages using the project level API:

$ cat .npmrc 
registry=http://gdk.test:8000/api/v4/projects/56/packages/npm/
//gdk.test:8000/api/v4/projects/56/packages/npm/:_authToken=XXXXXXXX

$ npm install @root/scoped-package-with-naming-convention
npm notice created a lockfile as package-lock.json. You should commit this file.
npm WARN pullfrominstance@1.0.0 No description
npm WARN pullfrominstance@1.0.0 No repository field.

+ @root/scoped-package-with-naming-convention@5.7.9
added 1 package from 1 contributor in 0.623s

$ npm install @any_scope/scoped_package                  
npm WARN pullfrominstance@1.0.0 No description
npm WARN pullfrominstance@1.0.0 No repository field.

+ @any_scope/scoped_package@5.7.9
added 1 package from 1 contributor in 0.912s

$ npm install non-scoped-package
npm WARN pullfrominstance@1.0.0 No description
npm WARN pullfrominstance@1.0.0 No repository field.

+ non-scoped-package@5.7.9
added 1 package from 1 contributor in 0.432s

$ npm dist-tag add non-scoped-package@5.7.9 my-tag
+my-tag: non-scoped-package@5.7.9

$ npm dist-tag ls non-scoped-package 
latest: 5.7.9
my-tag: 5.7.9

$ npm dist-tag rm non-scoped-package@5.7.9 my-tag
-my-tag: non-scoped-package@5.7.9

All commands are working as expected including $ dist-tag. 🎉

Instance level

Let's try now to pull those 3 packages using the instance level endpoint:

$ cat .npmrc  
registry=http://gdk.test:8000/api/v4/packages/npm/
//gdk.test:8000/api/v4/packages/npm/:_authToken=XXXXXXXXXXXXX

$ npm install @root/scoped-package-with-naming-convention
npm notice created a lockfile as package-lock.json. You should commit this file.
npm WARN pullfrominstance@1.0.0 No description
npm WARN pullfrominstance@1.0.0 No repository field.

+ @root/scoped-package-with-naming-convention@5.7.9
added 1 package from 1 contributor in 0.443s

$ npm install @any_scope/scoped_package
npm ERR! code E404
npm ERR! 404 Not Found - GET https://registry.npmjs.org/@any_scope/scoped_package - Not found
npm ERR! 404 
npm ERR! 404  '@any_scope/scoped_package@latest' is not in the npm registry.
npm ERR! 404 You should bug the author to publish it (or use the name yourself!)
npm ERR! 404 
npm ERR! 404 Note that you can also install from a
npm ERR! 404 tarball, folder, http url, or git url.

$ npm install non-scoped-package
npm ERR! code E404
npm ERR! 404 Not Found - GET https://registry.npmjs.org/non-scoped-package - Not found
npm ERR! 404 
npm ERR! 404  'non-scoped-package@latest' is not in the npm registry.
npm ERR! 404 You should bug the author to publish it (or use the name yourself!)
npm ERR! 404 
npm ERR! 404 Note that you can also install from a
npm ERR! 404 tarball, folder, http url, or git url.

$ npm dist-tag add @root/scoped-package-with-naming-convention@5.7.9 bananas
+bananas: @root/scoped-package-with-naming-convention@5.7.9

$ npm dist-tag ls @root/scoped-package-with-naming-convention              
bananas: 5.7.9
latest: 5.7.9

$ npm dist-tag rm @root/scoped-package-with-naming-convention@5.7.9 bananas
-bananas: @root/scoped-package-with-naming-convention@5.7.9

All commands for the properly scoped package, including $ dist-tag. 🎉

We can't install a package that is not following the naming convention. This is the expected behavior.

Notice that I have the NPM request forward enabled on my instance, so when I try to install a package not following the naming convention, I get redirected to npmjs.org.

⚙ Does this MR meet the acceptance criteria?

Conformity

Availability and Testing

Security

If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:

  • [-] Label as security and @ mention @gitlab-com/gl-security/appsec
  • [-] The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
  • [-] Security reports checked/validated by a reviewer from the AppSec team

💾 Database

Explain plans for the updated finder class are here.

Edited by David Fernandez

Merge request reports