Skip to content

Nuget - metadata extraction

David Fernandez requested to merge 36502-nuget-package-metadata-extraction-job-db-changes-and-dependencies into master

What does this MR do?

This MR adds the metadata extraction worker for the NuGet API.

Related issue #36502 (closed) and #20050 (closed)

This MR doesn't need documentation or a change log entry.

This MR is gated behind this feature flag (scoped by project): nuget_package_registry as this MR is part of the nuget MVC (See the epic: &2271 (closed)).

This MR adds the following:

  • Packages::Nuget::ExtractionWorker Metadata extraction worker. Using the services below, the worker will try to download the package file, unzip it, read the nuspec file and get the package name and version. Once it has those, it will update Packages::Package object and Packages::PackageFile accordingly.
  • The worker will use a new namespace package_repositories. As we expand support for other packages managers and packages features, we will be adding workers. This namespace will regroup all workers dealing with packages repositories.
  • Packages::Nuget::MetadataExtractionService. The service that takes a Packages::PackageFile as an input and try to download the archive, read the nuspec file and extract a fixed set of metadata fields: for this current MR, package_name and package_version.
  • Packages::Nuget::PackageFilenameService. This service will take a package name + version and create a proper .nupkg filename. NuGet package names are usually in CamelCase whereas the filepath name is in lowercase. This service seems like an overkill but it will be useful for an upcoming MR where nuget will ask for a package file using its filename.

This MR also fixes the following bugs:

  • Previously when uploading a nuget package a fixed package name and a fixed version were set. Those are updated by the worker described above. The issue is that if we have a second upload, it will use the same fixed package name and version and this will lead to a database insert failure: there is a unique index on project_id, package_name, package_version. To fix that, the package version has a fixed prefix and a random uuid.
  • Nuget uploads were not properly handled by workhorse. This has been fixed and the uploaded file is now received by rails under the package param name (and not file as previously).

Additional notes

  • If there is anything wrong during the metadata extraction (package archive can't be unzipped or nuspec file is not present within the archive), the package is left as it is and the worker will stop its job. We will probably need a cleanup worker for nuget packages where the metadata extraction failed.

Screenshots

Uploading a package with nuget:

$ nuget push DummyProject.DummyPackage.1.0.0.nupkg  -source "locally"
WARNING: No API Key was provided and no API Key could be found for 'https://gitlab.local:4443/api/v4/projects/19/packages/nuget'. To save an API Key for a source use the 'setApiKey' command.
Pushing DummyProject.DummyPackage.1.0.0.nupkg to 'https://gitlab.local:4443/api/v4/projects/19/packages/nuget'...
  PUT https://gitlab.local:4443/api/v4/projects/19/packages/nuget/
  Created https://gitlab.local:4443/api/v4/projects/19/packages/nuget/ 26190ms
Your package was pushed.

will trigger the background worker: Screenshot_2020-01-15_at_12.08.10

Once the worker has finished it's job, the package is visible in the UI: Screenshot_2020-01-15_at_12.08.50

Does this MR meet the acceptance criteria?

Conformity

Availability and Testing

Security

If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:

  • Label as security and @ mention @gitlab-com/gl-security/appsec
  • The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
  • Security reports checked/validated by a reviewer from the AppSec team
Edited by Tim Rizzi

Merge request reports