GitLab doesn't handle "host" when sending "endpoint" to Workhorse
Summary
When using GitLab to upload artifacts to S3 on an S3-compatible provider that isn't Amazon AWS, some requests will actually be sent to s3.amazonaws.com
instead of the S3 endpoint defined in the gitlab.rb
file.
Steps to reproduce
-
Use the following configuration in GitLab
13.8.x
:gitlab_rails['object_store']['enabled'] = true gitlab_rails['object_store']['connection'] = { 'provider' => 'AWS', 'region' => 'eu-central-1', 'aws_access_key_id' => '<redacted>', 'aws_secret_access_key' => '<redacted>', 'host' => 's3.eu-central-1.wasabisys.com' } #gitlab_rails['object_store']['storage_options'] = {} gitlab_rails['object_store']['proxy_download'] = false gitlab_rails['object_store']['objects']['artifacts']['bucket'] = '<redacted>' gitlab_rails['object_store']['objects']['external_diffs']['bucket'] = '<redacted>' gitlab_rails['object_store']['objects']['lfs']['bucket'] = '<redacted>' gitlab_rails['object_store']['objects']['uploads']['bucket'] = '<redacted>' gitlab_rails['object_store']['objects']['packages']['bucket'] = '<redacted>' gitlab_rails['object_store']['objects']['dependency_proxy']['enabled'] = false gitlab_rails['object_store']['objects']['dependency_proxy']['bucket'] = '<redacted>' gitlab_rails['object_store']['objects']['terraform_state']['bucket'] = '<redacted>' gitlab_rails['object_store']['objects']['pages']['bucket'] = '<redacted>'
-
Attempt to upload artifacts and observe the following errors in the logs:
{"error":"InvalidAccessKeyId: The AWS Access Key Id you provided does not exist in our records.\n\tstatus code: 403, request id: <redacted>, host id: <redacted>","level":"error","msg":"error uploading S3 session","time":"2021-02-01T22:41:28Z"} {"client_mode":"s3","copied_bytes":5242880,"correlation_id":"<redacted>","is_local":false,"is_multipart":false,"is_remote":true,"level":"info","msg":"saved file","remote_id":"1612219288-8133-0001-9710-79a8f12c74a1e29c41b229ece296e88e","remote_temp_object":"tmp/uploads/1612219288-8133-0001-9710-79a8f12c74a1e29c41b229ece296e88e","temp_file_prefix":"artifacts.zip","time":"2021-02-01T22:41:28Z"} {"correlation_id":"01EXFW42BSCQ12HK7EA9JPSTCH","level":"info","msg":"gitlab-zip-metadata error: zip archive not found, code: 12","time":"2021-02-01T22:41:28Z"}
As discussed with @stanhu in https://gitlab.slack.com/archives/C4Y5DRKLK/p1612315066194500?thread_ts=1612309852.181400&cid=C4Y5DRKLK, Rails doesn't handle the host
parameter when sending the endpoint
to Workhorse at the moment.
A workaround could be to try specifying the endpoint
instead and remove the host
like so:
gitlab_rails['object_store']['connection'] = {
'provider' => 'AWS',
'region' => 'eu-central-1',
'aws_access_key_id' => '<redacted>',
'aws_secret_access_key' => '<redacted>',
'host' => 's3.eu-central-1.wasabisys.com',
'endpoint' => 'https://s3.eu-central-1.wasabisys.com'
}
Example Project
What is the current bug behavior?
Artifact uploads to S3-compatible providers sometimes don't work.
What is the expected correct behavior?
Artifact uploads to S3-compatible providers should always work.
Relevant logs and/or screenshots
Output of checks
Results of GitLab environment info
Expand for output related to GitLab environment info
(For installations with omnibus-gitlab package run and paste the output of: `sudo gitlab-rake gitlab:env:info`) (For installations from source run and paste the output of: `sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production`)
Results of GitLab application Check
Expand for output related to the GitLab application check
(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true
)(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true
)(we will only investigate if the tests are passing)