S3 artifacts provider on GitLab: ERROR: Downloading artifacts from coordinator... error couldn't execute GET against

Summary

I recently changed my Gitlab instance to use Minio to store artifacts.

When a CI job is ran, the artifacts are correctly uploaded to my Minio instance. However, when another stage attempts to download these artifacts, it get's an error.

Steps to reproduce

There are many moving pieces to this problem (GitLab EE instance, gitlab runner, etc). I'm hoping the logs provide enough insight as to what is happening.

Actual behavior

I get download errors for artifacts on CI jobs.

Expected behavior

I expected future CI jobs to download the artifacts correctly.

Relevant logs and/or screenshots

Running with gitlab-runner 10.4.0 (857480b6)
  on build-linux (80ab73ef)
Using Docker executor with image microsoft/aspnetcore-build:2.0.5-2.1.4 ...
Using docker image sha256:10edb6331b0f4f63d409539f2db866e05dc3a18bc048fbf72a1ccb1010e599ef for predefined container...
Pulling docker image microsoft/aspnetcore-build:2.0.5-2.1.4 ...
Using docker image microsoft/aspnetcore-build:2.0.5-2.1.4 ID=sha256:e9127190cede10068e3d413aa17861713c679ae8ea5685c28dea8b05790fa5f1 for build container...
Running on runner-80ab73ef-project-2-concurrent-0 via build-linux...
Fetching changes...
Removing src/MedXStorage.Web/wwwroot/css/
Removing src/MedXStorage.Web/wwwroot/js/
HEAD is now at 09cd679 Modified worklist stores to AND request identifier results. Added calls to worklist stores to return worklistdata objects and added support for Find to the worklist rest API
Checking out 09cd6790 as develop...
Skipping Git submodules setup
Checking cache for develop_2...
Successfully extracted cache
Downloading artifacts for version (1275)...
ERROR: Downloading artifacts from coordinator... error couldn't execute GET against https://lab.mydomain.com/api/v4/jobs/1275/artifacts: Get http://minio.mydomain.com:9000/development-173014-gitlab-artifacts/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35/2018_04_10/1275/223/artifacts.zip?X-Amz-Expires=600&X-Amz-Date=20180410T201950Z&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=key/20180410//s3/aws4_request&X-Amz-SignedHeaders=host&X-Amz-Signature=[FILTERED]  id=1275 token=BWP86GDD
WARNING: Retrying...                               
ERROR: Downloading artifacts from coordinator... error couldn't execute GET against https://lab.mydomain.com/api/v4/jobs/1275/artifacts: Get http://minio.mydomain.com:9000/development-173014-gitlab-artifacts/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35/2018_04_10/1275/223/artifacts.zip?X-Amz-Expires=600&X-Amz-Date=20180410T201951Z&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=key/20180410//s3/aws4_request&X-Amz-SignedHeaders=host&X-Amz-Signature=[FILTERED]  id=1275 token=BWP86GDD
WARNING: Retrying...                               
ERROR: Downloading artifacts from coordinator... error couldn't execute GET against https://lab.mydomain.com/api/v4/jobs/1275/artifacts: Get http://minio.mydomain.com:9000/development-173014-gitlab-artifacts/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35/2018_04_10/1275/223/artifacts.zip?X-Amz-Expires=600&X-Amz-Date=20180410T201952Z&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=key/20180410//s3/aws4_request&X-Amz-SignedHeaders=host&X-Amz-Signature=[FILTERED]  id=1275 token=BWP86GDD
FATAL: invalid argument                            
ERROR: Job failed: exit code 1

My GitLab configuration for artifacts:

gitlab_rails['artifacts_enabled'] = true
gitlab_rails['artifacts_object_store_enabled'] = true
gitlab_rails['artifacts_object_store_background_upload'] = true
gitlab_rails['artifacts_object_store_remote_directory'] = "mybucket"
gitlab_rails['artifacts_object_store_connection'] = {
  'provider' => 'AWS',
  'region' => '',
  'aws_access_key_id' => 'key',
  'aws_secret_access_key' => 'secret',
  'host' => 'minio.mydomain.com',
  'endpoint' => 'http://minio.mydomain.com:9000',
  'path_style' => true
}

I noticed that the url https://gitlab.mydomain.com/api/v4/jobs/1275/artifacts returns a redirect directly to the Minio instance. This is surprising, but it is what it is. I figured GitLab would proxy that data. I can visit that url directly and everything works. Visiting Minio directly, I see the artifacts and I can download them. I can also download the artifacts from my GitLab instance from the job that created them.

Given the revelation that you guys aren't acting as a proxy with the S3 provider, is it possible that the gitlab-runner it accessing the Minio instance before the data is available? Notice that I have artifacts_object_store_background_upload set to true. If I set this to false, that means that artifacts will be uploaded directly to Minio, making them available for immediate download. Could this be the problem?

Environment description

I'm using a self-hosted GitLab EE instance running 10.4.3 and gitlab-runner version 10.4.0.

Both instances are running on separate machines. I am using the docker executor for the gitlab runner.

Used GitLab Runner version

Version:      10.4.0
Git revision: 857480b6
Git branch:   10-4-stable
GO version:   go1.8.5
Built:        Mon, 22 Jan 2018 09:47:33 +0000
OS/Arch:      linux/amd64