Skip to content

Investigate:Publish/install Conan packages with only name/version

Context

As mentioned in the epic &7093 (closed), users of the Conan repository need the ability to publish and install packages using only name/version. As this is a big change to the current functionality, this issue will serve as an investigation issue.

Proposal

Investigate adding support for allowing the project level endpoint of Conan to support packages with only name/version.

Results of the investigation

Conan uses a variety of API endpoints to manage packages.

These endpoints often use the recipe in the URL: .../name/version/username/channel/...

🕯 Digging the source

Looking at the conan routes, we see there is no route specified that does not include the username and channel. I decided to dig into a specific route to see what happens when the recipe (or reference) is generated.

First, we can see that when generating the recipe from the reference, Conan simply has logic that says "if no username or channel exists, leave them off": https://github.com/conan-io/conan/blob/e39b07f6de617e1c79e6c2992808925998123e0e/conans/model/ref.py#L234

Next, looking at how an incoming request is parsed into a reference, we see that if a username and channel are blank, they are processed as Python None values: https://github.com/conan-io/conan/blob/e39b07f6de617e1c79e6c2992808925998123e0e/conans/model/ref.py#L191

The reference validation checks if username or channel is blank, but allows the reference to be valid if they are both blank: https://github.com/conan-io/conan/blob/e39b07f6de617e1c79e6c2992808925998123e0e/conans/model/ref.py#L199

🔬 Looking at name/version upload requests

Based on the source code, it seems that Conan will still use /name/version/username/channel in it's URLs but allow the last two to be blank. The source explicitly has a space for each one with a / between them, so I hypothesize that we will see some sort of blank value or space holder in the requests.

I create a package without a username/channel:

conan new Hello/0.1 -t
conan create .

Then I add a local remote:

conan remote add asdf http://gdk.test:3001/api/v4/projects/83/packages/conan

And now I can attempt to upload the package:

$ CONAN_LOGIN_USERNAME=root CONAN_PASSWORD=<pat> conan upload Hello/0.1 -r asdf
DEBUG :conan_api.py   [176]: INIT: Using config '/Users/steveabrams/.conan/conan.conf' [2021-11-04 13:06:42,441]
DEBUG :tracer.py      [156]: CONAN_API: upload(pattern=Hello/0.1,package=None,query=None,remote_name=asdf,all_packages=False,policy=None,confirm=False,retry=None,retry_wait=None,integrity_check=False,parallel_upload=False) [2021-11-04 13:06:42,443]
Are you sure you want to upload 'Hello/0.1' to 'asdf'? (yes/no): y
Uploading to remote 'asdf':
Uploading Hello/0.1 to remote 'asdf'
DEBUG :rest_client_common.py[160]: REST: ping: http://gdk.test:3001/api/v4/projects/83/packages/conan/v1/ping [2021-11-04 13:06:58,943]
DEBUG :rest_client.py [58]: REST: Cached capabilities for the remote: [] [2021-11-04 13:07:00,094]
DEBUG :rest_client_common.py[188]: REST: get: http://gdk.test:3001/api/v4/projects/83/packages/conan/v1/conans/Hello/0.1/_/_/digest [2021-11-04 13:07:00,094]
DEBUG :rest_client_common.py[30]: REST ERROR: <class 'conans.errors.RequestErrorException'> [2021-11-04 13:07:00,434]
ERROR: Hello/0.1: Upload recipe to 'asdf' failed: {"error":"package_username is invalid, package_channel is invalid"}. [Remote: asdf]

In the output, we can see the Conan client inserts _ characters where the username/channel should be in the request: http://gdk.test:3001/api/v4/projects/83/packages/conan/v1/conans/Hello/0.1/_/_/digest.

The initial request for the recipe manifest fails because the username and channel are invalid. This makes sense because we validate that both values are present in the model. There is also a NOT NULL constraint on the columns in the database:

gitlabhq_development# \d packages_conan_metadata
                                          Table "public.packages_conan_metadata"
      Column      │           Type           │ Collation │ Nullable │                       Default
══════════════════╪══════════════════════════╪═══════════╪══════════╪═════════════════════════════════════════════════════
 id               │ bigint                   │           │ not null │ nextval('packages_conan_metadata_id_seq'::regclass)
 package_id       │ bigint                   │           │ not null │
 created_at       │ timestamp with time zone │           │ not null │
 updated_at       │ timestamp with time zone │           │ not null │
 package_username │ character varying(255)   │           │ not null │
 package_channel  │ character varying(255)   │           │ not null │
Indexes:
    "packages_conan_metadata_pkey" PRIMARY KEY, btree (id)
    "index_packages_conan_metadata_on_package_id_username_channel" UNIQUE, btree (package_id, package_username, package_channel)
Foreign-key constraints:
    "fk_rails_8c68cfec8b" FOREIGN KEY (package_id) REFERENCES packages_packages(id) ON DELETE CASCADE

💡 Solution

At the high level, we need to update the API to accept empty values for package_username and package_channel for the project-level endpoints.

📝 Steps

  1. Decide if it is better to have a blank or NULL value for the username and channel.
  2. Depending on (1.), remove the NOT NULL constraint on the database and update the model validation to be optional.
  3. Update conan_endpoints.rb to make the username and channel optional params. Better yet, we should create param blocks in the project and group api files so the group can still require them.
  4. Update the Conan::ApiHelpers module to properly handle when the username and channel are not present for project-level packages.
  5. Update related services, finders, presenters, and entities to also handle when username and channel are not present for project-level packages.

We should be able to guard this change with a feature flag. There may be some refactoring needed in the Conan::ApiHelpers and if we remove the database constraint, that cannot be guarded directly.

I think the overall work is not too extensive and could be weighed as a 2, but this will be a somewhat large MR (or two) with a more thorough review process, and a feature flag rollout will follow, so I will maintain the weight of 3.

A few more unknowns and questions

These could be addressed and tested during the implementation, but just noting them to keep them in mind.

  • What happens if you install a package by name/version and there are only packages with full recipes in the registry?
  • What happens if you install a package by full recipe and there is only a name/version package in the registry?
Edited by Tim Rizzi