Conan C/C++ Package Manager MVC
Problem to solve
Conan is an open source C/C++ package manager that was designed to help developers create and share their binaries. By integrating with Conan, GitLab will provide a centralized location to store and view those binaries, in the same place as their source code and pipelines. For customers building high performance software, including in the financial industry, we're seeing significant usage of C/C++ and we want to ensure that we are supporting these kinds of users as well.
This issue will contribute to the vision of the GitLab Package stage to provide a single interface for managing dependencies, registries, and package repositories.
Proposal for MVC
We are going to solve this problem by integrating with Conan, which will provide packaging support for users coding in C++. As a C/C++ developer, when I am creating, fetching or updating packages, I need a shared and secure environment to upload and view those packages, so that I can work collaboratively and efficiently with my team.
- We must allow, and provide documentation for, configuring the Conan client to add GitLab to the remote list.
- We must allow the user to authenticate with their personal access token
- Developers need the ability to create and upload packages to the GitLab remote
- All users (Reporter and up) need the ability to view their packages
These features offer parity with our other Package Manager integrations, such as Maven and NPM. The user interface can be exactly what we have now for Maven and NPM. The only change would be slightly different meta data, when drilling into each package. And if we are limited in our ability to change that, I can limit the fields to 3 to match our current design patterns.
As an MVC, we plan to limit the scope in several ways. First, we will focus first on allowing users to push dependencies to their project and allow for pulling dependencies at the Instance level, similar to how we implemented Maven. Based on customer feedback, instance level is the top priority. We can also limit the scope by limiting authentication to personal access tokens and create a separate issue for supporting CI_JOB_TOKEN.
Core Conan commands
The critical functionality Conan commands that impact us are:
conan remoteManages the remote list and the package recipes associated to a remote.
conan uploadUploads a recipe and binary packages to a remote.
conan infoGets information about the dependency graph of a recipe
conan searchSearches package recipes and binaries in the local cache or in a remote.
conan installInstalls the requirements specified in a recipe
conan removeRemoves the matching packages and binaries from local cache or remote.
- Administrators can turn on/off the Conan Repository
- Administrators can configure their package manager integrations, including Conan, to update the storage path, use object storage and migrate local packages to object storage.
- Developers coding in C/C++ and using Conan can add GitLab to their Conan remote list.
- Developers can authenticate with their personal access token.
- Developers can configure their Conan integration to work on the project / instance level.
- Developers can use Conan to create, modify and upload their packages to GitLab.
- Users with access to a given project can view and pull packages from the GitLab Package Repository
- Users can view basic meta data about packages from within the GitLab UI. (Name, Version, Type, Created (date))
- Users can click on a package to view additional details about it. For example, package name, version and created date. In addition, other Conan meta data, such as; OS, architecture, build type, compiler, compiler version, compiler.libcxx, shared (true/false)
- Developers can delete packages from within the GitLab UI.
User interface and meta data
We will display the following meta data which is passed through from Conan:
- Build type
- Compiler version
- compiler.cppstd (C++ Standard)
- This is a user generated list of key, value pairs included in settings.yml
The Conan API: v1 vs v2
After speaking with the Conan team, we will use v1 of the API.
Response from Conan team when asked which version fo the API we should use:
Definitely v1. This is still the default for Conan clients, as Api V2 is for revisions, and that is opt-in, and it will be opt-in for a while. Some users are now starting to use revisions, but it was released very recently, and as it is change in the way of doing things, it will take some time for users to migrate. In any case, I'd suggest implementing the storage ready for V2, even if you still don't provide the API v2. That means to have a default revision "0", which is overwritten if the same package with same version is uploaded again (un-revisioned). You also don't need the revision index, everything can be hardcoded to use "0". That way you won't need to run a storage migration when you need to implement API v2.
Permissions and Security
The permissions should follow the same levels as NPM and Maven. At the instance level, any user can view and download packages. But, only developers with permission to the project can upload and delete them.
|Pull from Maven repository or NPM registry or Conan Repository||x||x||x||x|
|Publish to Maven repository or NPM registry or Conan Repository||x||x||x|
- We will need added to the docs at https://docs.gitlab.com/ee/user/project/packages/conan_repository.html that details how to enable, authenticate, and configure Conan to work with GitLab.
- We will need to ensure we add the Conan Repository to the packages administration docs here.
We will have to test the core functionality works:
- Feature is only available for premium/gold users
- Documentation is updated and accurate
- Users can publish packages to GitLab
- Packages show up accurately in the UI
- UI functions as expected
- No degradation in functionality in our Maven, NPM or Container registries
- Ability to connect to object storage
What does success look like, and how can we measure that?
Since this feature is being requested by several customers, our first success metric will be working with those customers to get started using the GitLab Conan Repository. Adoption of the feature will be our first key success metric.
As a metric, we will look at the
total # of eligible users / # of actual users. To start, we can limit the scope of that to the customers that have requested this feature. We will also conduct user research to discover why the feature has or has not been adopted.
What is the type of buyer?
This feature will be focused on Director and Executives, as it is a Premium/Silver and Ultimate/Gold feature. https://about.gitlab.com/handbook/ceo/pricing/#four-tiers
Links / references
- Conan docs
- Conan API v1
- Conan API v2
- Conan center JFrog partnered with Conan to create a docker hub similar product for Conan packages
- JFrog's Conan integration
- The Conan V1 API endpoints that will be implemented in this issue and it's sub-issues consist mostly of the endpoints found in the conan server v1 code, as well as the common endpoints