### Problem to solve In milestone %"12.9", GitLab launched a feature that, by default, when an NPM package is not found in the GitLab NPM Registry, the [request is forwarded to npmjs.com](https://docs.gitlab.com/ee/user/packages/npm_registry/#forwarding-requests-to-npmjsorg). The request is forwarded to and handled by npm. By only forwarding the request, GitLab does not have any insight into which package was fetched or it's corresponding metadata. This information would be useful in filtering out suspicious packages or in caching external dependencies. ### Intended users * [Delaney (Development Team Lead)](https://about.gitlab.com/handbook/marketing/product-marketing/roles-personas/#delaney-development-team-lead) * [Sasha (Software Developer)](https://about.gitlab.com/handbook/marketing/product-marketing/roles-personas/#sasha-software-developer) * [Devon (DevOps Engineer)](https://about.gitlab.com/handbook/marketing/product-marketing/roles-personas/#devon-devops-engineer) * [Sidney (Systems Administrator)](https://about.gitlab.com/handbook/marketing/product-marketing/roles-personas/#sidney-systems-administrator) ### User experience goal - The goal of this issue will be to expand the functionality of the npm request forwarding feature to allow for the storage of package information and metadata. In the future, this will allow us to add features like [flagging suspicious packages before they are downloaded](https://gitlab.com/gitlab-org/gitlab/-/issues/215393) or [caching packages to reduce external dependencies](). - In the interim, the goal of this issue will be to ensure there is no degradation in the user experience of the npm request forwarding feature. ### Proposal The npm request forwarding feature will proxy requests to npmjs.com, so that we can store package details and metadata for future use. #### Metadata As an MVC we will store the following metadata, which is the current metadata stored and exposed in the GitLab NPM Registry - name - version - dist-tags - dependencies - dependencies - devDependencies - bundleDependencies - peerDependencies - deprecated However, we'd like to store and report all of a given package's metadata: - description - keywords - homepage - bugs - license - author - contributors - repository - publishConfig - browser - bin - man - directories - directories.lib - directories.bin - directories.man - directories.doc - directories.example - directories.test - scripts - config§engines - engineStrict - os - cpu <!-- How are we going to solve the problem? Try to include the user journey! https://about.gitlab.com/handbook/journeys/#user-journey --> ### Further details #### Questions - Should there be any indication in the UI that a package was received from npmjs.com? - When is the metadata updated? - When is the cache updated? (future state, beyond the MVC) #### User flow ##### Package not found in GitLab registry but found in npm public registry 1. A developer using the GitLab NPM Registry submits an MR for a minor change. 1. Their CI pipeline will download the `stable` tag of an npm package. 1. The package was not found in the GitLab registry. 1. A request is passed to npmjs.com. 1. Count event via snowplow 1. npm receives the request 1. The package is found and delivered to GitLab 1. GitLab stores package metadata 1. GitLab caches package for 30 days 1. Count event via snowplow 1. Pipeline runs successfully ##### Package not found in npm public registry 1. A developer using the GitLab NPM Registry submits an MR for a minor change. 1. Their CI pipeline will download the `stable` tag of an npm package. 1. The package was not found in the GitLab registry. 1. A request is passed to npmjs.com. 1. Count event via snowplow 1. npm receives the request 1. The package is not found 1. Count event via snowplow 1. Pipeline runs fail with error ##### Package not found in npm public registry but found in the cache (future state) 1. A developer using the GitLab NPM Registry submits an MR for a minor change. 1. Their CI pipeline will download the `stable` tag of an npm package. 1. The package was not found in the GitLab registry. 1. A request is passed to npmjs.com. 1. Count event via snowplow 1. npm receives the request 1. The package is not found 1. Count event via snowplow 1. GitLab searches the 30-day cache for the package 1. The package is found in the cache 1. Count event via snowplow 1. Pipeline runs successfully ##### Package not found in npm public registry and not found in the cache (future state) 1. A developer using the GitLab NPM Registry submits an MR for a minor change. 1. Their CI pipeline will download the `stable` tag of an npm package. 1. The package was not found in the GitLab registry. 1. A request is passed to npmjs.com. 1. Count event via snowplow 1. npm receives the request 1. The package is not found 1. Count event via snowplow 1. GitLab searches the 30-day cache for the package 1. The package is not found in the cache 1. Count event via snowplow 1. Pipeline fails ### Permissions and Security - There are no permissions changes required for this change. ### Documentation - [Update npm metdata](https://docs.gitlab.com/ee/user/packages/npm_registry/#npm-dependencies-metadata) - [Update npm request forwarding docs](https://docs.gitlab.com/ee/user/packages/npm_registry/#forwarding-requests-to-npmjsorg) <!-- See the Feature Change Documentation Workflow https://docs.gitlab.com/ee/development/documentation/workflow.html#for-a-product-change * Add all known Documentation Requirements in this section. See https://docs.gitlab.com/ee/development/documentation/feature-change-workflow.html#documentation-requirements * If this feature requires changing permissions, update the permissions document. See https://docs.gitlab.com/ee/user/permissions.html --> ### Availability & Testing <!-- This section needs to be retained and filled in during the workflow planning breakdown phase of this feature proposal, if not earlier. What risks does this change pose to our availability? How might it affect the quality of the product? What additional test coverage or changes to tests will be needed? Will it require cross-browser testing? Please list the test areas (unit, integration and end-to-end) that needs to be added or updated to ensure that this feature will work as intended. Please use the list below as guidance. * Unit test changes * Integration test changes * End-to-end test change See the test engineering planning process and reach out to your counterpart Software Engineer in Test for assistance: https://about.gitlab.com/handbook/engineering/quality/test-engineering/#test-planning --> ### What does success look like, and how can we measure that? Success looks like we are able to expand the functionality of the npm request forwarding feature to store and update key metadata so that in the future we can flag suspicious activity and cache external dependencies. #### Metrics - Count the number of events in which a request was forwarded to npmjs.com. (Number of failed/successful requests) We can measure this by measuring the adoption of this feature, which will be ### What is the type of buyer? Since it is common practice to pull packages from the public npm registry, this impacts all buyers. However in the future, the ~"Category:Dependency Firewall" and ~"Category:Dependency Proxy" features will be focused on large enterprises. <!-- What is the buyer persona for this feature? See https://about.gitlab.com/handbook/marketing/product-marketing/roles-personas/buyer-persona/ In which enterprise tier should this feature go? See https://about.gitlab.com/handbook/product/pricing/#four-tiers --> ### Is this a cross-stage feature? - This feature is used frequently with GitLab CI and will enable us to implement the Dependency Firewall in the future, which has a lot of cross-over with Secure and Defend. ### Links / references - [npm metadata details](https://docs.npmjs.com/files/package.json) <!-- triage-serverless v3 PLEASE DO NOT REMOVE THIS SECTION --> *This page may contain information related to upcoming products, features and functionality. It is important to note that the information presented is for informational purposes only, so please do not rely on the information for purchasing or planning purposes. Just like with all projects, the items mentioned on the page are subject to change or delay, and the development, release, and timing of any products, features, or functionality remain at the sole discretion of GitLab Inc.* <!-- triage-serverless v3 PLEASE DO NOT REMOVE THIS SECTION -->