Nuget - metadata extraction
requested to merge 36502-nuget-package-metadata-extraction-job-db-changes-and-dependencies into master
What does this MR do?
This MR adds the metadata extraction worker for the NuGet API.
Related issue #36502 (closed) and #20050 (closed)
This MR doesn't need documentation or a change log entry.
This MR is gated behind this feature flag (scoped by project): nuget_package_registry
as this MR is part of the nuget MVC (See the epic: &2271 (closed)).
This MR adds the following:
-
Packages::Nuget::ExtractionWorker
Metadata extraction worker. Using the services below, the worker will try to download the package file, unzip it, read thenuspec
file and get the package name and version. Once it has those, it will updatePackages::Package
object andPackages::PackageFile
accordingly. - The worker will use a new namespace
package_repositories
. As we expand support for other packages managers and packages features, we will be adding workers. This namespace will regroup all workers dealing with packages repositories. -
Packages::Nuget::MetadataExtractionService
. The service that takes aPackages::PackageFile
as an input and try to download the archive, read thenuspec
file and extract a fixed set of metadata fields: for this current MR,package_name
andpackage_version
. -
Packages::Nuget::PackageFilenameService
. This service will take a package name + version and create a proper.nupkg
filename. NuGet package names are usually in CamelCase whereas the filepath name is in lowercase. This service seems like an overkill but it will be useful for an upcoming MR wherenuget
will ask for a package file using its filename.
This MR also fixes the following bugs:
- Previously when uploading a nuget package a fixed package name and a fixed version were set. Those are updated by the worker described above. The issue is that if we have a second upload, it will use the same fixed package name and version and this will lead to a database insert failure: there is a unique index on project_id, package_name, package_version. To fix that, the package version has a fixed prefix and a random uuid.
- Nuget uploads were not properly handled by workhorse. This has been fixed and the uploaded file is now received by rails under the
package
param name (and notfile
as previously).
Additional notes
- If there is anything wrong during the metadata extraction (package archive can't be unzipped or
nuspec
file is not present within the archive), the package is left as it is and the worker will stop its job. We will probably need a cleanup worker for nuget packages where the metadata extraction failed.
Screenshots
Uploading a package with nuget
:
$ nuget push DummyProject.DummyPackage.1.0.0.nupkg -source "locally"
WARNING: No API Key was provided and no API Key could be found for 'https://gitlab.local:4443/api/v4/projects/19/packages/nuget'. To save an API Key for a source use the 'setApiKey' command.
Pushing DummyProject.DummyPackage.1.0.0.nupkg to 'https://gitlab.local:4443/api/v4/projects/19/packages/nuget'...
PUT https://gitlab.local:4443/api/v4/projects/19/packages/nuget/
Created https://gitlab.local:4443/api/v4/projects/19/packages/nuget/ 26190ms
Your package was pushed.
will trigger the background worker:
Once the worker has finished it's job, the package is visible in the UI:
Does this MR meet the acceptance criteria?
Conformity
-
Changelog entry -
Documentation (if required) -
Code review guidelines -
Merge request performance guidelines -
Style guides -
Database guides -
Separation of EE specific content
Availability and Testing
-
Review and add/update tests for this feature/bug. Consider all test levels. See the Test Planning Process. -
Tested in all supported browsers
Security
If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:
-
Label as security and @ mention @gitlab-com/gl-security/appsec
-
The MR includes necessary changes to maintain consistency between UI, API, email, or other methods -
Security reports checked/validated by a reviewer from the AppSec team
Edited by Tim Rizzi