Investigate:Publish/install Conan packages with only name/version
Context
As mentioned in the epic &7093 (closed), users of the Conan repository need the ability to publish and install packages using only name/version. As this is a big change to the current functionality, this issue will serve as an investigation issue.
Proposal
Investigate adding support for allowing the project level endpoint of Conan to support packages with only name/version.
Results of the investigation
Conan uses a variety of API endpoints to manage packages.
These endpoints often use the recipe in the URL: .../name/version/username/channel/...
🕯 Digging the source
Looking at the conan routes, we see there is no route specified that does not include the username
and channel
. I decided to dig into a specific route to see what happens when the recipe (or reference) is generated.
First, we can see that when generating the recipe from the reference, Conan simply has logic that says "if no username or channel exists, leave them off": https://github.com/conan-io/conan/blob/e39b07f6de617e1c79e6c2992808925998123e0e/conans/model/ref.py#L234
Next, looking at how an incoming request is parsed into a reference, we see that if a username and channel are blank, they are processed as Python None
values: https://github.com/conan-io/conan/blob/e39b07f6de617e1c79e6c2992808925998123e0e/conans/model/ref.py#L191
The reference validation checks if username or channel is blank, but allows the reference to be valid if they are both blank: https://github.com/conan-io/conan/blob/e39b07f6de617e1c79e6c2992808925998123e0e/conans/model/ref.py#L199
🔬 Looking at name/version upload requests
Based on the source code, it seems that Conan will still use /name/version/username/channel
in it's URLs but allow the last two to be blank. The source explicitly has a space for each one with a /
between them, so I hypothesize that we will see some sort of blank value or space holder in the requests.
I create a package without a username/channel:
conan new Hello/0.1 -t
conan create .
Then I add a local remote:
conan remote add asdf http://gdk.test:3001/api/v4/projects/83/packages/conan
And now I can attempt to upload the package:
$ CONAN_LOGIN_USERNAME=root CONAN_PASSWORD=<pat> conan upload Hello/0.1 -r asdf
DEBUG :conan_api.py [176]: INIT: Using config '/Users/steveabrams/.conan/conan.conf' [2021-11-04 13:06:42,441]
DEBUG :tracer.py [156]: CONAN_API: upload(pattern=Hello/0.1,package=None,query=None,remote_name=asdf,all_packages=False,policy=None,confirm=False,retry=None,retry_wait=None,integrity_check=False,parallel_upload=False) [2021-11-04 13:06:42,443]
Are you sure you want to upload 'Hello/0.1' to 'asdf'? (yes/no): y
Uploading to remote 'asdf':
Uploading Hello/0.1 to remote 'asdf'
DEBUG :rest_client_common.py[160]: REST: ping: http://gdk.test:3001/api/v4/projects/83/packages/conan/v1/ping [2021-11-04 13:06:58,943]
DEBUG :rest_client.py [58]: REST: Cached capabilities for the remote: [] [2021-11-04 13:07:00,094]
DEBUG :rest_client_common.py[188]: REST: get: http://gdk.test:3001/api/v4/projects/83/packages/conan/v1/conans/Hello/0.1/_/_/digest [2021-11-04 13:07:00,094]
DEBUG :rest_client_common.py[30]: REST ERROR: <class 'conans.errors.RequestErrorException'> [2021-11-04 13:07:00,434]
ERROR: Hello/0.1: Upload recipe to 'asdf' failed: {"error":"package_username is invalid, package_channel is invalid"}. [Remote: asdf]
In the output, we can see the Conan client inserts _
characters where the username/channel should be in the request: http://gdk.test:3001/api/v4/projects/83/packages/conan/v1/conans/Hello/0.1/_/_/digest
.
The initial request for the recipe manifest fails because the username and channel are invalid. This makes sense because we validate that both values are present in the model. There is also a NOT NULL
constraint on the columns in the database:
gitlabhq_development# \d packages_conan_metadata
Table "public.packages_conan_metadata"
Column │ Type │ Collation │ Nullable │ Default
══════════════════╪══════════════════════════╪═══════════╪══════════╪═════════════════════════════════════════════════════
id │ bigint │ │ not null │ nextval('packages_conan_metadata_id_seq'::regclass)
package_id │ bigint │ │ not null │
created_at │ timestamp with time zone │ │ not null │
updated_at │ timestamp with time zone │ │ not null │
package_username │ character varying(255) │ │ not null │
package_channel │ character varying(255) │ │ not null │
Indexes:
"packages_conan_metadata_pkey" PRIMARY KEY, btree (id)
"index_packages_conan_metadata_on_package_id_username_channel" UNIQUE, btree (package_id, package_username, package_channel)
Foreign-key constraints:
"fk_rails_8c68cfec8b" FOREIGN KEY (package_id) REFERENCES packages_packages(id) ON DELETE CASCADE
💡 Solution
At the high level, we need to update the API to accept empty values for package_username
and package_channel
for the project-level endpoints.
📝 Steps
- Decide if it is better to have a blank or NULL value for the username and channel.
- Depending on (1.), remove the
NOT NULL
constraint on the database and update the model validation to be optional. - Update conan_endpoints.rb to make the username and channel optional params. Better yet, we should create param blocks in the project and group api files so the group can still require them.
- Update the Conan::ApiHelpers module to properly handle when the username and channel are not present for project-level packages.
- Update related services, finders, presenters, and entities to also handle when username and channel are not present for project-level packages.
We should be able to guard this change with a feature flag. There may be some refactoring needed in the Conan::ApiHelpers
and if we remove the database constraint, that cannot be guarded directly.
I think the overall work is not too extensive and could be weighed as a 2
, but this will be a somewhat large MR (or two) with a more thorough review process, and a feature flag rollout will follow, so I will maintain the weight of 3
.
A few more unknowns and questions
These could be addressed and tested during the implementation, but just noting them to keep them in mind.
- What happens if you install a package by
name/version
and there are only packages with full recipes in the registry? - What happens if you install a package by full recipe and there is only a
name/version
package in the registry?