GitLab doesn't handle "host" when sending "endpoint" to Workhorse

Summary

When using GitLab to upload artifacts to S3 on an S3-compatible provider that isn't Amazon AWS, some requests will actually be sent to s3.amazonaws.com instead of the S3 endpoint defined in the gitlab.rb file.

Steps to reproduce

1. Use the following configuration in GitLab 13.8.x:

   gitlab_rails['object_store']['enabled'] = true
   gitlab_rails['object_store']['connection'] = {
           'provider' => 'AWS',
           'region' => 'eu-central-1',
           'aws_access_key_id' => '<redacted>',
           'aws_secret_access_key' => '<redacted>',
           'host' => 's3.eu-central-1.wasabisys.com'
   }
   #gitlab_rails['object_store']['storage_options'] = {}
   gitlab_rails['object_store']['proxy_download'] = false
   gitlab_rails['object_store']['objects']['artifacts']['bucket'] = '<redacted>'
   gitlab_rails['object_store']['objects']['external_diffs']['bucket'] = '<redacted>'
   gitlab_rails['object_store']['objects']['lfs']['bucket'] = '<redacted>'
   gitlab_rails['object_store']['objects']['uploads']['bucket'] = '<redacted>'
   gitlab_rails['object_store']['objects']['packages']['bucket'] = '<redacted>'
   
   gitlab_rails['object_store']['objects']['dependency_proxy']['enabled'] = false
   gitlab_rails['object_store']['objects']['dependency_proxy']['bucket'] = '<redacted>'
   
   gitlab_rails['object_store']['objects']['terraform_state']['bucket'] = '<redacted>'
   gitlab_rails['object_store']['objects']['pages']['bucket'] = '<redacted>'

2. Attempt to upload artifacts and observe the following errors in the logs:

   {"error":"InvalidAccessKeyId: The AWS Access Key Id you provided does not exist in our records.\n\tstatus code: 403, request id: <redacted>, host id: <redacted>","level":"error","msg":"error uploading S3 session","time":"2021-02-01T22:41:28Z"}
   {"client_mode":"s3","copied_bytes":5242880,"correlation_id":"<redacted>","is_local":false,"is_multipart":false,"is_remote":true,"level":"info","msg":"saved file","remote_id":"1612219288-8133-0001-9710-79a8f12c74a1e29c41b229ece296e88e","remote_temp_object":"tmp/uploads/1612219288-8133-0001-9710-79a8f12c74a1e29c41b229ece296e88e","temp_file_prefix":"artifacts.zip","time":"2021-02-01T22:41:28Z"}
   {"correlation_id":"01EXFW42BSCQ12HK7EA9JPSTCH","level":"info","msg":"gitlab-zip-metadata error: zip archive not found, code: 12","time":"2021-02-01T22:41:28Z"}

As discussed with @stanhu in https://gitlab.slack.com/archives/C4Y5DRKLK/p1612315066194500?thread_ts=1612309852.181400&cid=C4Y5DRKLK, Rails doesn't handle the host parameter when sending the endpoint to Workhorse at the moment.

A workaround could be to try specifying the endpoint instead and remove the host like so:

gitlab_rails['object_store']['connection'] = {
    'provider' => 'AWS',
    'region' => 'eu-central-1',
    'aws_access_key_id' => '<redacted>',
    'aws_secret_access_key' => '<redacted>',
    'host' => 's3.eu-central-1.wasabisys.com',
    'endpoint' => 'https://s3.eu-central-1.wasabisys.com'
}

Example Project

What is the current bug behavior?

Artifact uploads to S3-compatible providers sometimes don't work.

What is the expected correct behavior?

Artifact uploads to S3-compatible providers should always work.

Relevant logs and/or screenshots

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info

(For installations with omnibus-gitlab package run and paste the output of:
`sudo gitlab-rake gitlab:env:info`)

(For installations from source run and paste the output of:
`sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production`)

Results of GitLab application Check

Expand for output related to the GitLab application check

(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true)
(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true)
(we will only investigate if the tests are passing)

Possible fixes