Skip to content

Conan packages scoped to project-level

Steve Abrams requested to merge 11679-conan-project-level into master

What does this MR do?

NOTE: This MR has over 1000 lines added. Don't panic! The reason is a 900+ line spec file was translated to a shared_examples file so it could be used in two different places. See the notes on that spec towards the bottom of this description.

A little background on what this is all about 🔍

Instance-level packages

A package belongs to a project.

This MR implements project-level packages for the Conan package registry. Currently, GitLab has instance-level packages.

What this means is that right now, when you publish a package to GitLab, you use a remote url that is common to the entire GitLab instance: conan remote add gitlab https://gitlab.com/api/v4/packages/conan. When Conan sends the package to GitLab, we rely on the package name to tell us what project we should relate the package to. This means we have to force users to name their packages in a way that uniquely identifies the project.

The same goes for installing a package from the GitLab registry, GitLab only receives the name of the package, so it has to identify the project it's saved in in order to check permissions on the user making the request.

Project-level packages

If we add a set of routes that include the project ID, a user can set their remote with conan remote add gitlab https://gitlab.com/api/v4/projects/<project_id>/packages/conan. This means when they publish or install a package, GitLab does not need to identify the project from the package name, but can find it directly from the id in the requests. When this is the case, we can allow users to name their packages anything they'd like! 🎉

Back to what this MR does ⚙

This MR implements project-level packages by:

  1. Rename the current conan_packages.rb API file to conan_package_endpoints.rb and turn it into an ActiveSupport::Concern module that can be included in other API files.
  2. Create conan_instance_packages.rb and conan_project_packages.rb and wrap ConanPackageEndpoints inside of each respective namespace. This is to allow us to use the same endpoints with different prefixes.
  • Instance-level endpoints have the prefix api/v4/packages/conan/v1/
  • Project-level endpoints have the prefix api/v4/projects/:id/packages/conan/v1/
  1. Rename conan_packages_spec.rb to conan_instance_packages_spec.rb and create conan_project_packages_spec.rb (a few notes on this decision below).
  2. A few of the Conan API endpoints returns a set of URLs for the Conan client to use to upload or download the individual package files. We update the code in the Conan::PackagePresenter and Conan::ApiHelpers so it uses the correct Grape path helper to build the urls with the correct prefix (instance or project level).
  3. Update the docs to describe how to use the project-level remote and update the naming restriction content.

Notes on the specs

Because we now have two sets of API endpoints that do essentially the same thing, but with different url prefixes, we can mostly use the existing instance-level tests for both. Here is the approach I took

  • You cannot have dynamic data in the describe descriptions, so I decided to leave a describe for each endpoint in each spec file.
  • From there, I attempted to keep each endpoint as small as possible, using shared_examples so that both the project-level and instance-level run the same tests, but just feed them different URLs for the API calls.
  • The only other major difference is that when testing for a "not found project", the project-level tests must base this on an invalid project_id param, and the instance-level tests must base this on an invalid package_username param.
  • So although there is a lot of changes in the test code here, the conan_packages_shared_examples.rb and conan_packages_shared_context.rb files don't really contain any new code, just code migrated from the previous conan_packages_spec.rb file.

I'm happy to discuss if you have any questions here!

Screenshots 📷

Uploading a package with a project level remote

$ conan remote add localproject http://localhost:3001/api/v4/projects/9/packages/conan
$ CONAN_LOGIN_USERNAME=root CONAN_PASSWORD=<redacted> conan upload Hello/0.1@testname/stable --all -r localproject
DEBUG :conan_api.py   [163]: INIT: Using config '/Users/steveabrams/.conan/conan.conf' [2020-08-20 14:51:16,883]
DEBUG :tracer.py      [155]: CONAN_API: upload(pattern=Hello/0.1@testname/stable,package=None,query=None,remote_name=localproject,all_packages=True,policy=None,confirm=False,retry=None,retry_wait=None,integrity_check=False) [2020-08-20 14:51:16,884]
Uploading to remote 'localproject':
Uploading Hello/0.1@testname/stable to remote 'localproject'
DEBUG :rest_client_common.py[160]: REST: ping: http://localhost:3001/api/v4/projects/9/packages/conan/v1/ping [2020-08-20 14:51:16,901]
DEBUG :rest_client.py [38]: REST: Cached capabilities for the remote: [] [2020-08-20 14:51:17,300]
DEBUG :rest_client_common.py[185]: REST: get: http://localhost:3001/api/v4/projects/9/packages/conan/v1/conans/Hello/0.1/testname/stable/digest [2020-08-20 14:51:17,300]
DEBUG :rest_client_common.py[36]: REST ERROR: <class 'conans.errors.NotFoundException'> [2020-08-20 14:51:17,572]
DEBUG :rest_client_common.py[152]: REST: Check credentials: http://localhost:3001/api/v4/projects/9/packages/conan/v1/users/check_credentials [2020-08-20 14:51:17,572]
DEBUG :rest_client_common.py[185]: REST: get: http://localhost:3001/api/v4/projects/9/packages/conan/v1/conans/Hello/0.1/testname/stable [2020-08-20 14:51:17,887]
DEBUG :rest_client_common.py[179]: REST: post: http://localhost:3001/api/v4/projects/9/packages/conan/v1/conans/Hello/0.1/testname/stable/upload_urls [2020-08-20 14:51:18,125]
Uploading conanfile.py: 100%|##########| 1.72k/1.72k [00:00<00:00, 872kB/s]
Uploading conanmanifest.txt: 100%|##########| 58.0/58.0 [00:00<00:00, 30.6kB/s]
DEBUG :rest_client_v1.py[161]: UPLOAD:
                               All uploaded! Total time: 2.153989315032959
                                [2020-08-20 14:51:20,528]

Uploaded conan recipe 'Hello/0.1@testname/stable' to 'localproject': http://localhost:3001/api/v4/projects/9/packages/conan
Uploading package 1/1: 103f6067a947f366ef91fc1b7da351c588d1827f to 'localproject'
DEBUG :uploader.py    [353]: UPLOAD: Time remote_manager build_files_set : 0.001200 [2020-08-20 14:51:20,530]
DEBUG :rest_client_common.py[152]: REST: Check credentials: http://localhost:3001/api/v4/projects/9/packages/conan/v1/users/check_credentials [2020-08-20 14:51:20,532]
DEBUG :rest_client_common.py[185]: REST: get: http://localhost:3001/api/v4/projects/9/packages/conan/v1/conans/Hello/0.1/testname/stable/packages/103f6067a947f366ef91fc1b7da351c588d1827f [2020-08-20 14:51:20,942]
Requesting upload urls...                                             DEBUG :rest_client_common.py[179]: REST: post: http://localhost:3001/api/v4/projects/9/packages/conan/v1/conans/Hello/0.1/testname/stable/packages/103f6067a947f366ef91fc1b7da351c588d1827f/upload_urls [2020-08-20 14:51:21,176]
Requesting upload urls...Done!
Uploading conan_package.tgz: 100%|##########| 2.08k/2.08k [00:00<00:00, 965kB/s]
Uploading conaninfo.txt: 100%|##########| 440/440 [00:00<00:00, 223kB/s]
Uploading conanmanifest.txt: 100%|##########| 158/158 [00:00<00:00, 76.9kB/s]
DEBUG :rest_client_v1.py[161]: UPLOAD:
                               All uploaded! Total time: 3.1885528564453125
                                [2020-08-20 14:51:24,618]
DEBUG :uploader.py    [291]: UPLOAD: Time upload package: 4.089219 [2020-08-20 14:51:24,619]
DEBUG :uploader.py    [300]: UPLOAD: Time uploader upload_package: 4.090056 [2020-08-20 14:51:24,619]
DEBUG :uploader.py    [91]: UPLOAD: Time manager upload: 7.733696 [2020-08-20 14:51:24,620]

Screenshot of the package showing in the project

Screen_Shot_2020-08-21_at_4.50.03_PM

Does this MR meet the acceptance criteria?

Conformity

Availability and Testing

Security

If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:

  • [-] Label as security and @ mention @gitlab-com/gl-security/appsec
  • [-] The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
  • [-] Security reports checked/validated by a reviewer from the AppSec team

Related #11679 (closed)

Edited by Steve Abrams

Merge request reports