Can't upload Conan packages to object storage when using storage specific configuration

Summary

Tested on 13.7.1-ee

If you try to conan upload into the package registry for GitLab under the following conditions:

  1. You are using object storage for packages;
  2. You are using Amazon S3;
  3. You have direct_upload enabled;
  4. You are using storage specific configuration and not consolidated object storage configuration;

Then the conan upload will fail with the following:

$ CONAN_LOGIN_USERNAME=ci_user CONAN_PASSWORD=${CI_JOB_TOKEN} conan upload omega/$(cat VERSION.txt)@${CI_PROJECT_NAMESPACE////+}+${CI_PROJECT_NAME}/testing --all --remote=gitlab
DEBUG :conan_api.py   [177]: INIT: Using config '/home/conan/.conan/conan.conf' [2021-01-19 19:36:09,192]
DEBUG :tracer.py      [157]: CONAN_API: upload(pattern=omega/7.0@alpha+beta+gamma/testing,package=None,query=None,remote_name=gitlab,all_packages=True,policy=None,confirm=False,retry=None,retry_wait=None,integrity_check=False,parallel_upload=False) [2021-01-19 19:36:09,193]
Uploading to remote 'gitlab':
Uploading omega/7.0@alpha+beta+gamma/testing to remote 'gitlab'                            
DEBUG :rest_client_common.py[160]: REST: ping: https://<a mystery gitlab>/api/v4/packages/conan/v1/ping [2021-01-19 19:36:09,217]
Please log in to "gitlab" to perform this action. Execute "conan user" command.
DEBUG :rest_client_common.py[30]: REST ERROR: <class 'conans.errors.AuthenticationException'> [2021-01-19 19:36:09,254]
Remote 'gitlab' username: Got username 'ci_user' from environment
Please enter a password for "ci_user" account: Got password '******' from environment
DEBUG :rest_client_common.py[160]: REST: ping: https://<a mystery gitlab>/api/v4/packages/conan/v1/ping [2021-01-19 19:36:09,255]
DEBUG :rest_client.py [58]: REST: Cached capabilities for the remote: [] [2021-01-19 19:36:09,283]
DEBUG :rest_client_common.py[94]: REST: Authenticate to get access_token: https://<a mystery gitlab>/api/v4/packages/conan/v1/users/authenticate [2021-01-19 19:36:09,284]
DEBUG :rest_client_common.py[188]: REST: get: https://<a mystery gitlab>/api/v4/packages/conan/v1/conans/omega/7.0/alpha+beta+gamma/testing/digest [2021-01-19 19:36:09,323]
DEBUG :rest_client_common.py[30]: REST ERROR: <class 'conans.errors.NotFoundException'> [2021-01-19 19:36:09,401]
DEBUG :rest_client_common.py[152]: REST: Check credentials: https://<a mystery gitlab>/api/v4/packages/conan/v1/users/check_credentials [2021-01-19 19:36:09,403]
DEBUG :rest_client_common.py[188]: REST: get: https://<a mystery gitlab>/api/v4/packages/conan/v1/conans/omega/7.0/alpha+beta+gamma/testing [2021-01-19 19:36:09,433]
DEBUG :rest_client_common.py[182]: REST: post: https://<a mystery gitlab>/api/v4/packages/conan/v1/conans/omega/7.0/alpha+beta+gamma/testing/upload_urls [2021-01-19 19:36:09,488]
Uploading conanfile.py -> omega/7.0@alpha+beta+gamma/testing
Uploading conanmanifest.txt -> omega/7.0@alpha+beta+gamma/testing
ERROR: 
Error uploading file: conanfile.py, 'Internal server error
'
ERROR: 
Error uploading file: conanmanifest.txt, 'Internal server error
'
ERROR: omega/7.0@alpha+beta+gamma/testing: Upload recipe to 'gitlab' failed: Execute upload again to retry upload the failed files: conanfile.py, conanmanifest.txt. [Remote: gitlab]
ERROR: Errors uploading some packages
Cleaning up file based variables
00:01
ERROR: Job failed: exit code 1

Interestingly, while you are unable to upload Conan packages when you using storage specific configuration, you are able to upload Maven packages. This seems to suggest that Conan package uploads are handled differently in the code.

Steps to reproduce

  1. In your /etc/gitlab/gitlab.rb, ensure that you are not using consolidated object storage configuration and and configure storage specific configuration with direct_upload enabled. Run sudo gitlab-ctl reconfigure.

    gitlab_rails['packages_enabled'] = true
gitlab_rails['packages_object_store_enabled'] = true
gitlab_rails['packages_object_store_direct_upload'] = true
gitlab_rails['packages_object_store_background_upload'] = false
gitlab_rails['packages_object_store_proxy_download'] = false
gitlab_rails['packages_object_store_remote_directory'] = "xxx"
gitlab_rails['packages_object_store_connection'] = {
  'provider' => 'AWS',
  'region' => 'xxx',
  'aws_access_key_id' => 'xxx',
  'aws_secret_access_key' => 'xxx'
}

  2. Attempt to run conan upload. You will see errors in the output like:

    Error uploading file: conanfile.py, 'Internal server error
Error uploading file: conanmanifest.txt, 'Internal server error
ERROR: omega/7.0@alpha+beta+gamma/testing: Upload recipe to 'gitlab' failed: Execute upload again to retry upload the failed files: conanfile.py, conanmanifest.txt. [Remote: gitlab]
ERROR: Errors uploading some packages

    The Workhorse server logs /var/log/gitlab/gitlab-workhorse/current will indicate a 501 Not implemented error that is coming from Amazon S3:

    {
   "correlation_id":"01EWE2BDJE9332CPCQ5ZA7MSZY",
   "error":"BodyUploader: upload failed: PUT request https://<SANITIZED>.s3.ap-southeast-2.amazonaws.com/tmp/uploads/1611084969-7607-0001-2885-0ccd8ab7cb7465597f0c556aec6915fe?X-Amz-Expires=15300\u0026X-Amz-Date=20210119T193609Z\u0026X-Amz-Algorithm=AWS4-HMAC-SHA256\u0026X-Amz-Credential=<SANITIZED>%2F20210119%2Fap-southeast-2%2Fs3%2Faws4_request\u0026X-Amz-SignedHeaders=host\u0026X-Amz-Signature=[FILTERED] returned: 501 Not Implemented",
   "level":"error",
   "method":"PUT",
   "msg":"error",
   "time":"2021-01-19T19:36:09Z",
   "uri":"/api/v4/packages/conan/v1/files/omega/7.0/alpha+beta+gamma/testing/0/export/conanfile.py"
}
{
   "correlation_id":"01EWE2BE06P968KA9RC7RAEDPS",
   "error":"BodyUploader: upload failed: PUT request https://<SANITIZED>.s3.ap-southeast-2.amazonaws.com/tmp/uploads/1611084970-7609-0001-2750-2ba57dc4450eeef5173605ff794b217c?X-Amz-Expires=15300\u0026X-Amz-Date=20210119T193610Z\u0026X-Amz-Algorithm=AWS4-HMAC-SHA256\u0026X-Amz-Credential=<SANITIZED>%2F20210119%2Fap-southeast-2%2Fs3%2Faws4_request\u0026X-Amz-SignedHeaders=host\u0026X-Amz-Signature=[FILTERED] returned: 501 Not Implemented",
   "level":"error",
   "method":"PUT",
   "msg":"error",
   "time":"2021-01-19T19:36:10Z",
   "uri":"/api/v4/packages/conan/v1/files/omega/7.0/alpha+beta+gamma/testing/0/export/conanmanifest.txt"
}

Example Project

To aid with reproducing this bug, you can clone https://github.com/conan-io/hello and use the CI snippet in publish a Conan package by using CI/CD

What is the current bug behavior?

conan upload does not work if you are using storage specific configuration.

What is the expected correct behavior?

conan upload should work if you are using storage specific configuration.

Relevant logs and/or screenshots

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info 

(For installations with omnibus-gitlab package run and paste the output of:
`sudo gitlab-rake gitlab:env:info`)

(For installations from source run and paste the output of:
`sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production`)

Results of GitLab application Check

Expand for output related to the GitLab application check 
(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true)


(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true)


(we will only investigate if the tests are passing)

Possible fixes

Users should migrate to the consolidated object storage configuration to avoid this problem in the interim.

Edited by Anton Smith