Skip to content

Add worker to upload npm packages async

Moaz Khalifa requested to merge 433009-upload-npm-packages-async into master

What does this MR do and why?

Problem

Due to the upload of NPM packages happening synchronously we face some limitations and aren't able to parse the package tarball.

This leads to missing features like the support for overrides

Additionally, the async upload will bring to us a few other benefits https://gitlab.com/gitlab-org/gitlab/-/issues/417975#note_1601484030

Solution

Change the NPM package upload from synchronous to asynchronous.

Implementation

In this MR, we introduce a new background worker that should be responsible for handling the async upload flow.

To move from sync to async upload, we will mark the status of the uploaded package as processing. This means the package will appear in the UI, but it's not ready to be used yet till its status changes from processing to default.

The status update process will happen in the background job.

This worker will fetch the package file from the database using its id, and then pass the file to a service. The service is going to do two things:

  • download the file from the Object Storage and extract the package.json from it.
  • update the package's status from processing to default once the package.json is found.

This is a simplified flow and should be followed by more MRs to parse the package.json as intended.

The changes are gated behind a feature flag since this shift in the upload flow is a breaking change and should be released in milestone 17.

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Screenshots or screen recordings

Screenshots are required for UI changes, and strongly recommended for all other merge requests.

Before After

How to set up and validate locally

  1. Enable the feature flag in rails console:
    ::Feature.enable(:upload_npm_packages_async)
  2. Publish an NPM package. I used https://gitlab.com/10io/gl_pru#npm.
  3. The package should be published successfully with deafult status.

Related to #433009 (closed)

Edited by Moaz Khalifa

Merge request reports