**Problem:** Trying to publish a Conan or NPM Package, client will get "Internal server 500" error from Gitlab API/Response. **Current Condition:** Gitlab installed with Omnibus on Ubuntu on AWS EC2 - GitLab 13.3.4 (aebc07f48e8) - **Continuous Upgrades on Development Environment System Since Version 12.7** - GitLab Shell 13.6.0 - GitLab Workhorse v8.39.0 ** Packages & Registry Config in /etc/gitlab/gitlab.rb ** ** Note: Due to the original install being from Version 12.7, I wonder if this configuration is out of date? The "EE Only" comment referencing Maven gives it away.** ``` ################################################################################ ## Package repository (EE Only) ##! Docs: https://docs.gitlab.com/ee/administration/maven_packages.md ################################################################################ gitlab_rails['packages_enabled'] = true # gitlab_rails['packages_storage_path'] = "/var/opt/gitlab/gitlab-rails/shared/packages" gitlab_rails['packages_object_store_enabled'] = true gitlab_rails['packages_object_store_direct_upload'] = true # gitlab_rails['packages_object_store_background_upload'] = true gitlab_rails['packages_object_store_proxy_download'] = false gitlab_rails['packages_object_store_remote_directory'] = "acme-gitlab-packages" gitlab_rails['packages_object_store_connection'] = { 'provider' => 'AWS', 'region' => 'eu-west-2', 'aws_access_key_id' => 'AKIA................', 'aws_secret_access_key' => 'PvrM6...................................' # # # The below options configure an S3 compatible host instead of AWS # # 'host' => 's3.amazonaws.com', # # 'aws_signature_version' => 4, # For creation of signed URLs. Set to 2 if provider does not support v4. # # 'endpoint' => 'https://s3.amazonaws.com', # default: nil - Useful for S3 compliant services such as DigitalOcean Spaces # # 'path_style' => false # Use 'host/bucket_name/object' instead of 'bucket_name.host/object' } ``` **Technical Details: ** Looking at the logs from 'gitlab-ctl tail' , it seems workhorse is getting a 501 from S3 API. ``` ==> /var/log/gitlab/gitlab-rails/api_json.log <== {"time":"2020-09-16T18:40:36.527Z","severity":"INFO","duration_s":0.02567,"db_duration_s":0.00559,"view_duration_s":0.02008,"status":200,"method":"PUT","path":"/api/v4/packages/conan/v1/files/Hello/0.1/grouptest+projecttest/beta/0/export/conanfile.py/authorize","params":[],"host":"gitlab.dev.acme.com","remote_ip":"204.115.1XX.XXX, 10.0.2.206","ua":"Conan/1.29.0 (Python 2.7.16) python-requests/2.21.0","route":"/api/:version/packages/conan/v1/files/:package_name/:package_version/:package_username/:package_channel/:recipe_revision/export/:file_name/authorize","user_id":2,"username":"smithj","redis_calls":3,"redis_duration_s":0.000577,"redis_read_bytes":33,"redis_write_bytes":333,"redis_cache_calls":3,"redis_cache_duration_s":0.000577,"redis_cache_read_bytes":10,"redis_cache_write_bytes":217,"redis_shared_state_read_bytes":23,"redis_shared_state_write_bytes":116,"correlation_id":"929afaab-23fa-48ec-85f1-98b5265fdfa1","meta.user":"smithj","meta.project":"grouptest/projecttest","meta.root_namespace":"grouptest","meta.caller_id":"/api/:version/packages/conan/v1/files/:package_name/:package_version/:package_username/:package_channel/:recipe_revision/export/:file_name/authorize"} ==> /var/log/gitlab/gitlab-workhorse/current <== {"copied_bytes":0,"correlation_id":"ZWIX9WKgcC4","is_local":false,"is_multipart":false,"is_remote":true,"level":"info","msg":"saved file","remote_id":"1600281636-28853-0002-9330-bc7f78d05932dff51122d7f29d3ee3df","temp_file_prefix":"","time":"2020-09-16T18:40:36Z","use_s3_client":false} {"correlation_id":"R6HJ9goRe4a","duration_ms":0,"host":"gitlab.dev.acme.com","level":"info","method":"POST","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"52.207.1XX.XXX:0","remote_ip":"52.207.1XX.XXX","status":204,"system":"http","time":"2020-09-16T18:40:36Z","uri":"/api/v4/jobs/request","user_agent":"gitlab-runner 13.3.0 (13-3-stable; go1.13.8; linux/amd64)","written_bytes":0} ==> /var/log/gitlab/nginx/gitlab_access.log <== 10.0.2.206 - - [16/Sep/2020:18:40:36 +0000] "POST /api/v4/jobs/request HTTP/1.1" 204 0 "" "gitlab-runner 13.3.0 (13-3-stable; go1.13.8; linux/amd64)" - ==> /var/log/gitlab/gitlab-workhorse/current <== {"correlation_id":"ZWIX9WKgcC4","error":"BodyUploader: upload failed: PUT request https://acme-gitlab-packages.s3-eu-west-2.amazonaws.com/tmp/uploads/1600281636-28853-0002-9330-bc7f78d05932dff51122d7f29d3ee3df?X-Amz-Expires=15300\u0026X-Amz-Date=20200916T184036Z\u0026X-Amz-Algorithm=AWS4-HMAC-SHA256\u0026X-Amz-Credential=AKIAQLJXS6PQRNOEQXXX%2F20200916%2Feu-west-2%2Fs3%2Faws4_request\u0026X-Amz-SignedHeaders=host\u0026X-Amz-Signature=[FILTERED] returned: 501 Not Implemented","level":"error","method":"PUT","msg":"error","time":"2020-09-16T18:40:36Z","uri":"/api/v4/packages/conan/v1/files/Hello/0.1/grouptest+projecttest/beta/0/export/conanfile.py"} {"correlation_id":"ZWIX9WKgcC4","duration_ms":367,"host":"gitlab.dev.acme.com","level":"info","method":"PUT","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"204.115.1XX.XXX:0","remote_ip":"204.115.1XX.XXX","status":500,"system":"http","time":"2020-09-16T18:40:36Z","uri":"/api/v4/packages/conan/v1/files/Hello/0.1/grouptest+projecttest/beta/0/export/conanfile.py","user_agent":"Conan/1.29.0 (Python 2.7.16) python-requests/2.21.0","written_bytes":22} ==> /var/log/gitlab/nginx/gitlab_access.log <== 10.0.2.206 - - [16/Sep/2020:18:40:36 +0000] "PUT /api/v4/packages/conan/v1/files/Hello/0.1/grouptest+projecttest/beta/0/export/conanfile.py HTTP/1.1" 500 22 "" "Conan/1.29.0 (Python 2.7.16) python-requests/2.21.0" - ``` ** Some Background/Thoughts ** https://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html#ErrorCodeList The Link to AWS Docs for the API imply that "a header you provided implies functionality that is not implemented." This error is also occurring with NPM packages as well, so I think it is specific to the workhorse service, as described here: [https://docs.gitlab.com/ee/development/uploads.html#direct-upload](https://docs.gitlab.com/ee/development/uploads.html#direct-upload) I should add that my Gitlab Omnibus install is using external_url and HTTPS Proxy to HTTP, (e.g. 443 external from ELB, 80 internal to NGINX). I&#39;ve configured the gitlab.rb appropriately and S3 Backups and S3 LFS Objects are working without any issue. Thanks for your time.