Conan Upload API Endpoints
Implement the 7 API endpoints (details below) needed in order to successfully run conan upload
and have a conan package uploaded to GitLab. It will be stored using object storage similar to Maven packages.
This issue description contains the technical details uncovered while learning how the Conan API works.
This is the 3rd of 6 steps to implementing the Conan API https://gitlab.com/gitlab-org/gitlab-ee/issues/8248:
- #12567 (closed) - Skeleton API - Ping endpoint
- #12568 (closed) - Conan authentication endpoints
-
#13345 (closed) -
conan upload
endpoints -
#13350 (closed)
conan install
endpoints -
#13347 (closed)
conan search
endpoints -
conan info
endpoints (all endpoints forconan info
are implemented in previous steps) -
#31941 (closed)
conan remove
endpoints
Endpoints to implement in this issue:
-
GET /v1/users/check_credentials
- common_check_credentials
-
GET /v1/conans/{name}/{version}/{username}/{channel}
- recipe snapshot -
GET /v1/conans/{name}/{version}/{username}/{channel}/digest
- recipe digest -
POST /v1/conans/{name}/{version}/{username}/{channel}/upload_urls
- recipe upload urls
-
GET /v1/conans/{name}/{version}/{username}/{channel}/packages/{package_id}
- package snapshot -
GET /v1/conans/{name}/{version}/{username}/{channel}/packages/{package_id}/digest
- package digest -
POST /v1/conans/{name}/{version}/{username}/{channel}/packages/{package_id}/upload_urls
- package upload urls
-
PUT /v1/files/{recipe}/-/{path}/authorize
- handle the pre-auth call to workhorse for object storage acceleration. -
PUT /v1/files/{recipe}/-/{path}
- upload file
conan upload
makes a series of API calls to the registry (aka remote). Here is an example:
GET http://server:9300/v1/ping # implemented in #12567
GET http://server:9300/v1/conans/Hello/1.0/user/channel/digest
GET http://server:9300/v1/users/authenticate # implemented in #12568
GET http://server:9300/v1/conans/Hello/1.0/user/channel/digest
GET http://server:9300/v1/users/check_credentials
GET http://server:9300/v1/conans/Hello/1.0/user/channel
POST http://server:9300/v1/conans/Hello/1.0/user/channel/upload_urls
PUT http://server:9300/v1/files/Hello/1.0/user/channel/0/export/conanfile.py?signature=**********
PUT http://server:9300/v1/files/Hello/1.0/user/channel/0/export/conanmanifest.txt?signature=**********
GET http://server:9300/v1/users/check_credentials
GET http://server:9300/v1/conans/Hello/1.0/user/channel/packages/0ab9fcf606068d4347207cc29edd400ceccbc944
POST http://server:9300/v1/conans/Hello/1.0/user/channel/packages/0ab9fcf606068d4347207cc29edd400ceccbc944/upload_urls
PUT http://server:9300/v1/files/Hello/1.0/user/channel/0/package/0ab9fcf606068d4347207cc29edd400ceccbc944/0/conan_package.tgz?signature=**********
PUT http://server:9300/v1/files/Hello/1.0/user/channel/0/package/0ab9fcf606068d4347207cc29edd400ceccbc944/0/conaninfo.txt?signature=**********
PUT http://server:9300/v1/files/Hello/1.0/user/channel/0/package/0ab9fcf606068d4347207cc29edd400ceccbc944/0/conanmanifest.txt?signature=**********
All conan commands begin with the ping
endpoint to ensure connection to the remote. The upload follows the following process:
-
recipe digest
api call checks the remote for the recipe. If no recipe exists, 404 is returned, otherwise a hash of the recipe files and manifest are returned:
{
"conanmanifest.txt": "http://localhost:3001/{download_path_to_file_on_remote}",
"conanfile.py": "http://localhost:3001/{download_path_to_file_on_remote}"
}
-
If the files from the remote do not exist locally, they are downloaded using the returned urls
-
recipe snapshot
api call checks the recipe files md5 signatures returning a collection of md5 hashes. If the recipe does not exist an empty hash is returned.
{
"conanmanifest.txt": "{md5 hash of file on remote}",
"conanfile.py": "{md5 hash of file on remote}"
}
-
Client compares the returned hashes to diff the new recipe files with the existing ones in order to know which files it needs to request uploads for.
-
recipe upload urls
request is made supplying the api with a list of files that need to be uploaded (for a new package, it will be all files, for existing, it will be based on the diff performed). The client will request with a body like this:
{"conanfile.py": 1762, "conanmanifest.txt": 58}
The remote will respond with the upload urls for each file. The file size is given by the client so the remote can decide if it needs to offload the upload to a different service (all our uploads will be handled by workhorse so this should not matter, however, we should place limits on filesizes). The remote will return:
{
"conanmanifest.txt": "http://localhost:3001/{upload_path_for_file_on_remote}",
"conanfile.py": "http://localhost:3001/{upload_path_for_file_on_remote}"
}
-
The client will now use the returned upload urls to upload each file using the
upload file
endpoint. -
A similar process is followed for the package files and the package endpoints.