Skip to content

Build, publish and share packages to the GitLab NuGet (.NET) Repository

Problem to solve

.NET developers need a mechanism to create, share, and consume packages that contain compiled code and other content in projects that consume these packages. For .NET, the Microsoft-supported mechanism for sharing code is NuGet, which defines how packages for .NET are created, hosted, and consumed, and provides the tools for each of those roles.

By integrating with NuGet, GitLab will provide a centralized location to store and view those packages, in the same place as their source code and pipelines.

This issue will contribute to the vision of the GitLab Package stage to provide a single interface for managing dependencies, registries, and package repositories.

Intended users

Proposal

Provide support for users coding in .NET by integrating with NuGet and allowing developers to publish, share and consume .NET packages alongside their source code and CI/CD pipelines.

Proposed MVC Scope

  • We will provide documentation for, configuring the NuGet client to add GitLab to the remote list.
  • Users are able to add a project NuGet registry url to nuget in an authenticated manner using GitLab username + token.
  • Users are able to push a NuGet package (nuget push)
  • Users are able to pull a NuGet package (nuget install)
  • Users are able to delete a NuGet package (for the MVC, we will only allow deletion via the Packages API or the UI)
  • The MVC will be limited to the backend work and we will address the UI in subsequent milestones.

NuGet Commands (MVC)

Below is a list of the core NuGet commands that we will cover in this MVC.

The NuGet Server API defines 4+1 services as required: (services as in "nuget services". Nuget services can handle 1 or many urls)

  • Service Index
  • Push/Delete Service
  • Package Metadata Service
  • Package Content Service

Example UI (not included in the MVC)

NuGet_example

The Service Index
GET {api_base_url}/index.json
  • Simple json describing which services are available and what are their urls.
The Search Service (Not part of the MVC)
GET {search_service_url}?q={QUERY}&skip={SKIP}&take={TAKE}&prerelease={PRERELEASE}&semVerLevel={SEMVERLEVEL}
  • Parameters
    • q the search term. Server implementation is free to apply this filter on any field. Example: the search term can be applied to the name or the name and the description of a package, etc.
    • skip pagination parameter. This is the offset. Default to 0
    • take pagination parameter. This is the number of entries per page. Server implementation can impose a max value
    • prerelease to include prerelease packages or not
    • semVerLevel to include SemVer 2.0.0 packages
The Push/Delete Service
PUT {push_service_url}
  • The NuGet (compressed) archive is sent without any additional parameters.
  • ⚠ This means that this request has no metadata available. In short, we get a compressed archive and that's it.
  • ⚠ The backend will need to extract and inspect this compressed archive. Since this can take some time to do, this extraction should be done in a background job.

Log excerpt:

$ nuget push Bananas.1.0.0.nupkg -Source "local"
WARNING: No API Key was provided and no API Key could be found for 'http://localhost:4000/api/v4/projects/40/packages/nuget'. To save an API Key for a source use the 'setApiKey' command.
Pushing Bananas.1.0.0.nupkg to 'http://localhost:4000/api/v4/projects/40/packages/nuget'...
  PUT http://localhost:4000/api/v4/projects/40/packages/nuget/
  Created http://localhost:4000/api/v4/projects/40/packages/nuget/ 38ms
Your package was pushed.
DELETE {push_service_url}/{ID}/{VERSION}
  • Interpretation left to the server implementation
    • NuGet.org and github unlists the package but archives are still available for pulling.
    • Can be a hard delete. ⚠ This can easily break project builds.
POST {push_service_url}/{ID}/{VERSION}
  • Relist a previously unlisted package
  • No body request
The Package Metadata Service
GET {metadata_service_url}/{LOWER_ID}/index.json
GET {metadata_service_url}/{LOWER_ID}/{VERSION}.json
  • similar response as above but for a given version
  • undocumented but implemented by github
  • not sure that this url is used by nuget or Visual Studio
The Package Content Service
GET {content_service_url}/{LOWER_ID}/index.json
  • Simple json to with a single array: all available versions.
GET {content_service_url}/{LOWER_ID}/{LOWER_VERSION}/{LOWER_ID}.{LOWER_VERSION}.nupkg
  • The archive of the given package
  • Yes LOWER_ID and LOWER_VERSION are duplicated in the url

GET {content_service_url}/{LOWER_ID}/{LOWER_VERSION}/{LOWER_ID}.nuspec

  • The nuspec file of the archive
  • Not called during my tests but documented
  • Not implemented by github (404 Not Found)
Configuration and Authentication
  • The main authentication mechanism is a http basic auth but the Source/Feed has to be under https.
  • Should be compatible with GitLab username + token.
  • For the Push Service an additional authentication mechanism is available: an API token. This token is sent along the push request as a custom http header: X-NuGet-ApiKey. This could be used for the CI_JOB_TOKEN (see below for other future iterations)

Further details

What we are not doing in the MVC

There are a few things that we are not committing to with this MVC:

  • A front end for the NuGet Repository
  • Group/sub-group/instance level support

User stories

Administrator
  • I as an administrator of GitLab, need the ability to enable/disable the NuGet Repository, so that I can ensure the developers in my organization have access to the features that they are supposed to.
  • I as an administrator of Gitlab, need the ability to configure object storage for the GitLab Package Registry, including the NuGet Repository, so that I can optimize how my organization utilizes storage.
Developer
  • I as a developer, need the ability to configure NuGet to point to GitLab as a remote repository, so that I can push, pull and view my NuGet Packages with GitLab.
  • I as a developer, need the ability to setup authentication between GitLab and NuGet using my personal access token, so that I can push and pull packages to the GitLab NuGet Repository.
  • I as a developer, need the ability to run NuGet primary commands from the NuGet CLI to push, pull and update NuGet packages in the GitLab NuGet Repository at the instance and project level.
  • I as a developer, need the ability to view basic meta data about packages from within the GitLab UI, so that I can verify package info and ensure my project is using the correct dependencies.
  • I as a developer need the ability to view meta data from the .nuspec file associated with a package, so that I can understand how a package was built, by whom and when.
  • I as a developer, need the ability to delete packages from within the GitLab UI, so that I can remove old packages and ensure they are not accidentally used in my project.
Reporter
  • I as a project-stakeholder need the ability to view and pull packages from the NuGet Repository, so that I can view, inspect and download NuGet packages.

Naming conventions

Since the MVC scope is for the project level, we won't need a specific naming convention. However, for future versions (if we implement the instance level access for example), we can enforce the use of . as a separator to identify a Group: MyGroup.MyPackage.

Development pain points

  • The metadata of a package will need to be available in the database as it is heavily used by the Package Metadata Service.
  • The upload is a simple archive upload. No parameters about the metadata.
  • It looks like we will need to open the archive and extract the metadata to have it available within the GitLab database. This should be done in a background job.
  • Good news: the dependency models implemented in !14263 (closed) can be reused.
  • The Search Service uses a custom pagination parameters set. Those will need to be translated into what we use in the code.

Opportunities

  • We could implement this as an internal API as suggested in #35798 (closed)
  • The Metadata extraction job could be implemented in a generic way to be easily ported to other package types.

Proposed MRs breakdown

  1. API Skeleton + The Service Index + authentication
  2. Workhorse updates to handle the upload url
  3. The Push/Delete Service
  4. Nuget package metadata extraction job + db changes to support metadata and dependencies information
  5. The Package Metadata Service
  6. The Package Content Service
  7. The Search Service (Not part of the MVC)

The critical path would be: 1. 2. -> 3. -> 4. 5. 6. 7.

User Interface

The UI for this feature will be added in another issue. The below example is for providing clarity and may change in it's implementation.

Limitations

  • NuGet PackageReference is not available in Visual Studio 2015 and earlier. Migrated projects can be opened only in Visual Studio 2017 and later.
  • Migration is not currently available for C++ and ASP.NET projects.
  • Some packages may not be fully compatible with PackageReference. For more information, see package compatibility issues.

Permissions and Security

The permissions should follow the same levels as NPM, Maven and Conan.

Project Permissions: UI

Action Guest Reporter Developer Maintainer Owner
Pull from Maven, NPM, Conan, NuGet x x x x
Publish to Maven, NPM, Conan, NuGet x x x

Project Permissions: API

Action Guest Reporter Developer Maintainer Owner
List project packages (5) x x
Get a project package x x
List package files x x
Delete a project package x x

Group Permissions: API

Action Guest Reporter Developer Maintainer Owner
List the packages of a group (coming soon) x x

Instance Level Permissions

Action Guest Reporter Developer Maintainer Owner
Enable the Packages feature x
Migrate local packages to object storage x
Disable the Packages feature x

Documentation

What does success look like, and how can we measure that?

The goal of the Package Group is to ensure that in 3 years, 90% of our customers are using GitLab as their sole package registry. Success for this issue will be that we begin to empower .NET developers to utilize GitLab for publishing and sharing packages.

Outcomes

  • We'd like to see a 5% increase in overall adoption of the GitLab Package Registry with the launch of this MVC
  • We expect that number to rise to 10%, once we enable group/sub-group support and CI_JOB_TOKEN support
  • As this is an MVC, we expect to see an increased volume of issues for the feature.

Data and metrics

Priority Category Metric Aggregated by SMAU Eligible? Data available?
P1 Package Registry NuGet Registry enabled/disabled
  • Time: day/week/month/quarter/year
  • Instance: Self-managed CE/Self-managed EE
  • Account type: Core/Starter/Premium/Ultimate
 Yes No
P1 Package Registry # of page views of /packages
  • Time: day/week/month/quarter/year
  • Instance: Self-managed CE/Self-managed EE
  • Account type: Core/Starter/Premium/Ultimate
 Yes Yes
P1 Package Registry # of packages pushed to the NuGet Registry
  • Time: day/week/month/quarter/year
  • Instance: Self-managed CE/Self-managed EE
  • Account type: Core/Starter/Premium/Ultimate
  • Tool: Command Line/CI/CD
 Yes No
P2 Package Registry # of NuGet packages deleted
  • Time: day/week/month/quarter/year
  • Instance: Self-managed CE/Self-managed EE
  • Account type: Core/Starter/Premium/Ultimate
 Yes No
P2 Package Registry # of NuGet packages updated
  • Time: day/week/month/quarter/year
  • Instance: Self-managed CE/Self-managed EE
  • Account type: Core/Starter/Premium/Ultimate
 Yes No

Testing

  • Add Feed, push and pull from nuget (macOS + Windows)
  • Add Feed, pull and search from Visual Studio (macOS + Windows)
  • Push package from dotnet command (macOS + Windows)
  • (Bonus: have a quick check if chocolatey packages work with the MVC (Windows))

Future Development

Links / references

Edited by Tim Rizzi