Build, publish and share packages to the GitLab NuGet (.NET) Repository
Problem to solve
.NET developers need a mechanism to create, share, and consume packages that contain compiled code and other content in projects that consume these packages. For .NET, the Microsoft-supported mechanism for sharing code is NuGet, which defines how packages for .NET are created, hosted, and consumed, and provides the tools for each of those roles.
By integrating with NuGet, GitLab will provide a centralized location to store and view those packages, in the same place as their source code and pipelines.
This issue will contribute to the vision of the GitLab Package stage to provide a single interface for managing dependencies, registries, and package repositories.
Intended users
Proposal
Provide support for users coding in .NET by integrating with NuGet and allowing developers to publish, share and consume .NET packages alongside their source code and CI/CD pipelines.
Proposed MVC Scope
- We will provide documentation for, configuring the NuGet client to add GitLab to the remote list.
- Users are able to add a project NuGet registry url to
nuget
in an authenticated manner using GitLab username + token. - Users are able to push a NuGet package (
nuget push
) - Users are able to pull a NuGet package (
nuget install
) - Users are able to delete a NuGet package (for the MVC, we will only allow deletion via the Packages API or the UI)
- The MVC will be limited to the backend work and we will address the UI in subsequent milestones.
NuGet Commands (MVC)
Below is a list of the core NuGet commands that we will cover in this MVC.
The NuGet Server API defines 4+1 services as required: (services as in "nuget services". Nuget services can handle 1 or many urls)
- Service Index
- Push/Delete Service
- Package Metadata Service
- Package Content Service
Example UI (not included in the MVC)
The Service Index
- https://docs.microsoft.com/en-us/nuget/api/service-index
- Used as entry point when a private Source/Feed is added.
- Aggressively cached by clients
GET {api_base_url}/index.json
- Simple json describing which services are available and what are their urls.
The Search Service (Not part of the MVC)
- https://docs.microsoft.com/en-us/nuget/api/search-query-service-resource
- Used by Visual Studio. (search not available in
nuget
)
GET {search_service_url}?q={QUERY}&skip={SKIP}&take={TAKE}&prerelease={PRERELEASE}&semVerLevel={SEMVERLEVEL}
- Parameters
-
q
the search term. Server implementation is free to apply this filter on any field. Example: the search term can be applied to the name or the name and the description of a package, etc. -
skip
pagination parameter. This is the offset. Default to 0 -
take
pagination parameter. This is the number of entries per page. Server implementation can impose a max value -
prerelease
to include prerelease packages or not -
semVerLevel
to include SemVer 2.0.0 packages
-
The Push/Delete Service
- https://docs.microsoft.com/en-us/nuget/api/package-publish-resource
- Can be authenticated through a custom http header.
PUT {push_service_url}
- The NuGet (compressed) archive is sent without any additional parameters.
-
⚠ This means that this request has no metadata available. In short, we get a compressed archive and that's it. -
⚠ The backend will need to extract and inspect this compressed archive. Since this can take some time to do, this extraction should be done in a background job.
Log excerpt:
$ nuget push Bananas.1.0.0.nupkg -Source "local"
WARNING: No API Key was provided and no API Key could be found for 'http://localhost:4000/api/v4/projects/40/packages/nuget'. To save an API Key for a source use the 'setApiKey' command.
Pushing Bananas.1.0.0.nupkg to 'http://localhost:4000/api/v4/projects/40/packages/nuget'...
PUT http://localhost:4000/api/v4/projects/40/packages/nuget/
Created http://localhost:4000/api/v4/projects/40/packages/nuget/ 38ms
Your package was pushed.
DELETE {push_service_url}/{ID}/{VERSION}
- Interpretation left to the server implementation
- NuGet.org and github unlists the package but archives are still available for pulling.
- Can be a hard delete.
⚠ This can easily break project builds.
POST {push_service_url}/{ID}/{VERSION}
- Relist a previously unlisted package
- No body request
The Package Metadata Service
- https://docs.microsoft.com/en-us/nuget/api/registration-base-url-resource
- The most complex endpoint
- Not only metadata is listed but also
⚠ dependencies
GET {metadata_service_url}/{LOWER_ID}/index.json
- bulk listing of all metadata for all available versions
- the json for the metadata is quite complex (https://docs.microsoft.com/en-us/nuget/api/registration-base-url-resource#sample-response-1)
- can be broken into pages. (I didn't find an example for this :/)
- Uses the
LOWER_ID
andLOWER_VERSION
GET {metadata_service_url}/{LOWER_ID}/{VERSION}.json
- similar response as above but for a given version
- undocumented but implemented by github
- not sure that this url is used by
nuget
or Visual Studio
The Package Content Service
- https://docs.microsoft.com/en-us/nuget/api/package-base-address-resource
- Service to get the archive but also some metadata (see below)
- Uses the
LOWER_ID
andLOWER_VERSION
GET {content_service_url}/{LOWER_ID}/index.json
- Simple json to with a single array: all available versions.
GET {content_service_url}/{LOWER_ID}/{LOWER_VERSION}/{LOWER_ID}.{LOWER_VERSION}.nupkg
- The archive of the given package
- Yes LOWER_ID and LOWER_VERSION are duplicated in the url
GET {content_service_url}/{LOWER_ID}/{LOWER_VERSION}/{LOWER_ID}.nuspec
- The
nuspec
file of the archive - Not called during my tests but documented
- Not implemented by github (404 Not Found)
Configuration and Authentication
- The main authentication mechanism is a http basic auth but the Source/Feed has to be under https.
- Should be compatible with GitLab username + token.
- For the Push Service an additional authentication mechanism is available: an API token. This token is sent along the push request as a custom http header:
X-NuGet-ApiKey
. This could be used for the CI_JOB_TOKEN (see below for other future iterations)
Further details
What we are not doing in the MVC
There are a few things that we are not committing to with this MVC:
- A front end for the NuGet Repository
- Group/sub-group/instance level support
User stories
Administrator
- I as an administrator of GitLab, need the ability to enable/disable the NuGet Repository, so that I can ensure the developers in my organization have access to the features that they are supposed to.
- I as an administrator of Gitlab, need the ability to configure object storage for the GitLab Package Registry, including the NuGet Repository, so that I can optimize how my organization utilizes storage.
Developer
- I as a developer, need the ability to configure NuGet to point to GitLab as a remote repository, so that I can push, pull and view my NuGet Packages with GitLab.
- I as a developer, need the ability to setup authentication between GitLab and NuGet using my personal access token, so that I can push and pull packages to the GitLab NuGet Repository.
- I as a developer, need the ability to run NuGet primary commands from the NuGet CLI to push, pull and update NuGet packages in the GitLab NuGet Repository at the instance and project level.
- I as a developer, need the ability to view basic meta data about packages from within the GitLab UI, so that I can verify package info and ensure my project is using the correct dependencies.
- I as a developer need the ability to view meta data from the .nuspec file associated with a package, so that I can understand how a package was built, by whom and when.
- I as a developer, need the ability to delete packages from within the GitLab UI, so that I can remove old packages and ensure they are not accidentally used in my project.
Reporter
- I as a project-stakeholder need the ability to view and pull packages from the NuGet Repository, so that I can view, inspect and download NuGet packages.
Naming conventions
Since the MVC scope is for the project level, we won't need a specific naming convention. However, for future versions (if we implement the instance level access for example), we can enforce the use of .
as a separator to identify a Group: MyGroup.MyPackage
.
Development pain points
- The metadata of a package will need to be available in the database as it is heavily used by the Package Metadata Service.
- The upload is a simple archive upload. No parameters about the metadata.
- It looks like we will need to open the archive and extract the metadata to have it available within the GitLab database. This should be done in a background job.
- Good news: the dependency models implemented in !14263 (closed) can be reused.
- The Search Service uses a custom pagination parameters set. Those will need to be translated into what we use in the code.
Opportunities
- We could implement this as an internal API as suggested in #35798
- The Metadata extraction job could be implemented in a generic way to be easily ported to other package types.
Proposed MRs breakdown
- API Skeleton + The Service Index + authentication
- Workhorse updates to handle the upload url
- The Push/Delete Service
- Nuget package metadata extraction job + db changes to support metadata and dependencies information
- The Package Metadata Service
- The Package Content Service
- The Search Service (Not part of the MVC)
The critical path would be: 1. 2. -> 3. -> 4. 5. 6. 7.
User Interface
The UI for this feature will be added in another issue. The below example is for providing clarity and may change in it's implementation.
Limitations
- NuGet PackageReference is not available in Visual Studio 2015 and earlier. Migrated projects can be opened only in Visual Studio 2017 and later.
- Migration is not currently available for C++ and ASP.NET projects.
- Some packages may not be fully compatible with PackageReference. For more information, see package compatibility issues.
Permissions and Security
The permissions should follow the same levels as NPM, Maven and Conan.
Project Permissions: UI
Action | Guest | Reporter | Developer | Maintainer | Owner |
---|---|---|---|---|---|
Pull from Maven, NPM, Conan, NuGet | x | x | x | x | |
Publish to Maven, NPM, Conan, NuGet | x | x | x |
Project Permissions: API
Action | Guest | Reporter | Developer | Maintainer | Owner |
---|---|---|---|---|---|
List project packages (5) | x | x | |||
Get a project package | x | x | |||
List package files | x | x | |||
Delete a project package | x | x |
Group Permissions: API
Action | Guest | Reporter | Developer | Maintainer | Owner |
---|---|---|---|---|---|
List the packages of a group (coming soon) | x | x |
Instance Level Permissions
Action | Guest | Reporter | Developer | Maintainer | Owner |
---|---|---|---|---|---|
Enable the Packages feature | x | ||||
Migrate local packages to object storage | x | ||||
Disable the Packages feature | x |
Documentation
- Add a new section to the package registry user guides for the NuGet Repository at https://docs.gitlab.com/ee/user/project/packages/nuget_repository.html
- Packages API
- Packages Admin
What does success look like, and how can we measure that?
The goal of the Package Group is to ensure that in 3 years, 90% of our customers are using GitLab as their sole package registry. Success for this issue will be that we begin to empower .NET developers to utilize GitLab for publishing and sharing packages.
Outcomes
- We'd like to see a 5% increase in overall adoption of the GitLab Package Registry with the launch of this MVC
- We expect that number to rise to 10%, once we enable group/sub-group support and
CI_JOB_TOKEN
support - As this is an MVC, we expect to see an increased volume of issues for the feature.
Data and metrics
Priority | Category | Metric | Aggregated by | SMAU Eligible? | Data available? |
---|---|---|---|---|---|
P1 | Package Registry | NuGet Registry enabled/disabled |
|
Yes | No |
P1 | Package Registry | # of page views of /packages
|
|
Yes | Yes |
P1 | Package Registry | # of packages pushed to the NuGet Registry |
|
Yes | No |
P2 | Package Registry | # of NuGet packages deleted |
|
Yes | No |
P2 | Package Registry | # of NuGet packages updated |
|
Yes | No |
Testing
- Add Feed, push and pull from
nuget
(macOS + Windows) - Add Feed, pull and search from Visual Studio (macOS + Windows)
- Push package from
dotnet command
(macOS + Windows) - (Bonus: have a quick check if chocolatey packages work with the MVC (Windows))
Future Development
- NuGet Search Service
- NuGet Front End extracted metadata + dependencies integration (this might need some BE effort on the packages API)
- Authenticate with
CI_JOB_TOKEN
so that a CI job can push or update a package - Push/pull NuGet packages at the Group/sub-group level
- Push NuGet packages at the Instance level
- Add NuGet Packages to storage counter
- Metadata extraction job can lead to several interesting features:
- Files within the archive available for scanners (Vulnerabilites, Virus Scanner, others)
- Open this extraction to other packages type. Example, what if we expose the dependencies of an NPM package in the UI?
- Quotas and limits
Links / references
- Internal demo of Chocaltey
- Microsoft NuGet Docs
- NuGet v3 API
- https://api.nuget.org/v3/index.json
- Mono (for running nuget.exe on mac/linux)
- JFrog's NuGet Integration
- Sonatype's NuGet Integration
- Sleet: Several users have mentioned this project as a good example