Proxy and store package metadata for requests forwarded to npmjs.com

Problem to solve

In milestone %12.9, GitLab launched a feature that, by default, when an NPM package is not found in the GitLab NPM Registry, the request is forwarded to npmjs.com. The request is forwarded to and handled by npm.

By only forwarding the request, GitLab does not have any insight into which package was fetched or it's corresponding metadata. This information would be useful in filtering out suspicious packages or in caching external dependencies.

Intended users

• Delaney (Development Team Lead)
• Sasha (Software Developer)
• Devon (DevOps Engineer)
• Sidney (Systems Administrator)

User experience goal

• The goal of this issue will be to expand the functionality of the npm request forwarding feature to allow for the storage of package information and metadata. In the future, this will allow us to add features like flagging suspicious packages before they are downloaded or caching packages to reduce external dependencies.
• In the interim, the goal of this issue will be to ensure there is no degradation in the user experience of the npm request forwarding feature.

Proposal

The npm request forwarding feature will proxy requests to npmjs.com, so that we can store package details and metadata for future use.

Metadata

As an MVC we will store the following metadata, which is the current metadata stored and exposed in the GitLab NPM Registry

• name
• version
• dist-tags
• dependencies
• dependencies
  • devDependencies
  • bundleDependencies
  • peerDependencies
  • deprecated

However, we'd like to store and report all of a given package's metadata:

• description
• keywords
• homepage
• bugs
• license
• author
• contributors
• repository
• publishConfig
• browser
• bin
• man
• directories
• directories.lib
• directories.bin
• directories.man
• directories.doc
• directories.example
• directories.test
• scripts
• config§engines
• engineStrict
• os
• cpu

Further details

Questions

• Should there be any indication in the UI that a package was received from npmjs.com?
• When is the metadata updated?
• When is the cache updated? (future state, beyond the MVC)

User flow

Package not found in GitLab registry but found in npm public registry

1. A developer using the GitLab NPM Registry submits an MR for a minor change.
2. Their CI pipeline will download the stable tag of an npm package.
3. The package was not found in the GitLab registry.
4. A request is passed to npmjs.com.
5. Count event via snowplow
6. npm receives the request
7. The package is found and delivered to GitLab
8. GitLab stores package metadata
9. GitLab caches package for 30 days
10. Count event via snowplow
11. Pipeline runs successfully

Package not found in npm public registry

1. A developer using the GitLab NPM Registry submits an MR for a minor change.
2. Their CI pipeline will download the stable tag of an npm package.
3. The package was not found in the GitLab registry.
4. A request is passed to npmjs.com.
5. Count event via snowplow
6. npm receives the request
7. The package is not found
8. Count event via snowplow
9. Pipeline runs fail with error

Package not found in npm public registry but found in the cache (future state)

1. A developer using the GitLab NPM Registry submits an MR for a minor change.
2. Their CI pipeline will download the stable tag of an npm package.
3. The package was not found in the GitLab registry.
4. A request is passed to npmjs.com.
5. Count event via snowplow
6. npm receives the request
7. The package is not found
8. Count event via snowplow
9. GitLab searches the 30-day cache for the package
10. The package is found in the cache
11. Count event via snowplow
12. Pipeline runs successfully

Package not found in npm public registry and not found in the cache (future state)

1. A developer using the GitLab NPM Registry submits an MR for a minor change.
2. Their CI pipeline will download the stable tag of an npm package.
3. The package was not found in the GitLab registry.
4. A request is passed to npmjs.com.
5. Count event via snowplow
6. npm receives the request
7. The package is not found
8. Count event via snowplow
9. GitLab searches the 30-day cache for the package
10. The package is not found in the cache
11. Count event via snowplow
12. Pipeline fails

Permissions and Security

• There are no permissions changes required for this change.

Documentation

• Update npm metdata
• Update npm request forwarding docs

Availability & Testing

What does success look like, and how can we measure that?

Success looks like we are able to expand the functionality of the npm request forwarding feature to store and update key metadata so that in the future we can flag suspicious activity and cache external dependencies.

Metrics

• Count the number of events in which a request was forwarded to npmjs.com. (Number of failed/successful requests)

We can measure this by measuring the adoption of this feature, which will be

What is the type of buyer?

Since it is common practice to pull packages from the public npm registry, this impacts all buyers. However in the future, the Category:Dependency Firewall and Category:Dependency Proxy features will be focused on large enterprises.

Is this a cross-stage feature?

• This feature is used frequently with GitLab CI and will enable us to implement the Dependency Firewall in the future, which has a lot of cross-over with Secure and Defend.

Links / references

• npm metadata details

This page may contain information related to upcoming products, features and functionality. It is important to note that the information presented is for informational purposes only, so please do not rely on the information for purchasing or planning purposes. Just like with all projects, the items mentioned on the page are subject to change or delay, and the development, release, and timing of any products, features, or functionality remain at the sole discretion of GitLab Inc.