multipart artifact uploads fails with `timeout awaiting response headers`

Summary

multipart artifact uploads to Minio fail with timeout awaiting response headers.

direct_upload set to false and background_upload set to true results in artifacts being uploaded on gitlab's server (but I'm not sure if they end up being uploaded to minio at a later date)

Minio is using --compat

Steps to reproduce

Use an on-premise distributed instance of minio as an object store for large artifacts

What is the current bug behavior?

Artifact uploads fail

What is the expected correct behavior?

Artifact upload succeeds

Relevant logs and/or screenshots

{"correlation_id":"GcKAAGVxUr2","error":"handleFileUploads: extract files from multipart: persisting multipart file: upload part 4: PUT request \"https://minio.company.com/com.company.gitlab.artifacts/tmp/uploads/1565130885-25847-0001-5367-36f3f5928ecc63a858b7c2f8042e6ea4?uploadId=f41c8f6b-8237-4009-9577-70466f1e7d25\u0026partNumber=4\u0026X-Amz-Expires=15299\u0026X-Amz-Date=20190806T223446Z\u0026X-Amz-Algorithm=AWS4-HMAC-SHA256\u0026X-Amz-Credential=FILTERED%2F20190806%2Feu-east-1%2Fs3%2Faws4_request\u0026X-Amz-SignedHeaders=host\u0026X-Amz-Signature=[FILTERED]\": Put https://minio.company.com/com.company.gitlab.artifacts/tmp/uploads/1565130885-25847-0001-5367-36f3f5928ecc63a858b7c2f8042e6ea4?uploadId=f41c8f6b-8237-4009-9577-70466f1e7d25\u0026partNumber=4\u0026X-Amz-Expires=15299\u0026X-Amz-Date=20190806T223446Z\u0026X-Amz-Algorithm=AWS4-HMAC-SHA256\u0026X-Amz-Credential=FILTERED%2F20190806%2Feu-east-1%2Fs3%2Faws4_request\u0026X-Amz-SignedHeaders=host\u0026X-Amz-Signature=[FILTERED]: net/http: timeout awaiting response headers","level":"error","method":"POST","msg":"error","time":"2019-08-06T18:36:55-04:00","uri":"/api/v4/jobs/47733/artifacts?artifact_format=zip\u0026artifact_type=archive\u0026expire_in=1+day"}

On minio's side, it sees a bunch of s3.PutObjectParth, but at some point the following happens:

minio0 [REQUEST s3.AbortMultipartUpload] [22:36:55.085716620]
minio0 DELETE /com.company.gitlab.artifacts/tmp/uploads/1565130885-25847-0001-5367-36f3f5928ecc63a858b7c2f8042e6ea4?uploadId=f41c8f6b-8237-4009-9577-70466f1e7d25&X-Amz-Expires=15299&X-Amz-Date=20190806T223446Z&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=YP1A2T6K17SCL7T49R67%2F20190806%2Feu-east-1%2Fs3%2Faws4_request&X-Amz-SignedHeaders=host&X-Amz-Signature=164f6daff194c4dc59c9bac494fdd1ab1af9e646b6b93880c135629420294e36
minio0 Host: minio.company.com
minio0 Accept-Encoding: gzip
minio0 Content-Length: 0
minio0 User-Agent: Go-http-client/1.1
minio0 X-Forwarded-For: 192.168.4.165
minio0 
minio0 [RESPONSE] [22:36:55.123376561]
minio0 204 No Content
minio0 Server: MinIO/RELEASE.2019-08-01T22-18-54Z
minio0 Vary: Origin
minio0 X-Amz-Request-Id: 15B8765DA51C02D2
minio0 X-Xss-Protection: 1; mode=block
minio0 Accept-Ranges: bytes
minio0 Content-Length: 0
minio0 Content-Security-Policy: block-all-mixed-content
minio0 minio0 
minio2 [REQUEST s3.PutObjectPart] [22:35:56.800987758]
minio2 PUT /com.company.gitlab.artifacts/tmp/uploads/1565130885-25847-0001-5367-36f3f5928ecc63a858b7c2f8042e6ea4?uploadId=f41c8f6b-8237-4009-9577-70466f1e7d25&partNumber=4&X-Amz-Expires=15299&X-Amz-Date=20190806T223446Z&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=[FILTERED]%2F20190806%2Feu-east-1%2Fs3%2Faws4_request&X-Amz-SignedHeaders=host&X-Amz-Signature=[FILTERED]
minio2 Host: minio.company.com
minio2 User-Agent: Go-http-client/1.1
minio2 X-Forwarded-For: 192.168.4.165
minio2 X-Request-Id: GcKAAGVxUr2
minio2 Accept-Encoding: gzip
minio2 Content-Length: 104857600
minio2 <BODY>
minio2 [RESPONSE] [22:36:59.524203531]
minio2 404 Not Found
minio2 Content-Security-Policy: block-all-mixed-content
minio2 Content-Type: application/xml
minio2 Server: MinIO/RELEASE.2019-08-01T22-18-54Z
minio2 Vary: Origin
minio2 X-Amz-Request-Id: 15B876501313A0FE
minio2 X-Xss-Protection: 1; mode=block
minio2 Accept-Ranges: bytes
minio2 Content-Length: 575
minio2 <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>NoSuchUpload</Code><Message>The specified multipart upload does not exist. The upload ID may be invalid, or the upload may have been aborted or completed.</Message><Key>tmp/uploads/1565130885-25847-0001-5367-36f3f5928ecc63a858b7c2f8042e6ea4</Key><BucketName>com.company.gitlab.artifacts</BucketName><Resource>/com.company.gitlab.artifacts/tmp/uploads/1565130885-25847-0001-5367-36f3f5928ecc63a858b7c2f8042e6ea4</Resource><RequestId>15B876501313A0FE</RequestId><HostId>b2cbe4bd-4e41-4276-b487-8eab69be92d0</HostId></Error>minio2

For some reason gitlab times out after a few parts uploaded, tells minio to abort the multipart upload (rendering the upload ID invalid) and then gitlab tries to upload some parts again with the same upload ID.

I have tried uploading to minio using s3 cli (even with parallel upload set to 1), minio's CLI and curl and it all works fine, even on the same host as Gitlab is installed.

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info

System information System: Ubuntu 18.04 Current User: git Using RVM: no Ruby Version: 2.6.3p62 Gem Version: 2.7.9 Bundler Version:1.17.3 Rake Version: 12.3.2 Redis Version: 3.2.12 Git Version: 2.21.0 Sidekiq Version:5.2.7 Go Version: unknown

GitLab information Version: 12.1.4 Revision: 34d086f3e14 Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: PostgreSQL DB Version: 10.7 URL: https://gitlab.company.com HTTP Clone URL: https://gitlab.company.com/some-group/some-project.git SSH Clone URL: git@gitlab.company.com:some-group/some-project.git Using LDAP: yes Using Omniauth: yes Omniauth Providers:

GitLab Shell Version: 9.3.0 Repository storage paths:

default: /var/opt/gitlab/git-data/repositories GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell Git: /opt/gitlab/embedded/bin/git

Results of GitLab application Check

Expand for output related to the GitLab application check

Checking GitLab subtasks ...

Checking GitLab Shell ...

GitLab Shell: ... GitLab Shell version >= 9.3.0 ? ... OK (9.3.0) Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Check GitLab API access: OK Redis available via internal API: OK

Access to /var/opt/gitlab/.ssh/authorized_keys: OK gitlab-shell self-check successful

Checking GitLab Shell ... Finished

Checking Gitaly ...

Gitaly: ... default ... OK

Checking Gitaly ... Finished

Checking Sidekiq ...

Sidekiq: ... Running? ... yes Number of Sidekiq processes ... 1

Checking Sidekiq ... Finished

Checking Incoming Email ...

Incoming Email: ... Reply by email is disabled in config/gitlab.yml

Checking Incoming Email ... Finished

Checking LDAP ...

LDAP: ... Server: ldapmain LDAP authentication... Success LDAP users with access to your GitLab server (only showing the first 100 results) User output sanitized. Found 100 users of 100 limit.

Checking LDAP ... Finished

Checking GitLab App ...

Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ... 91/2 ... yes 93/6 ... yes 95/8 ... yes 97/10 ... yes 103/13 ... yes 105/14 ... yes 106/17 ... yes 105/18 ... yes 103/19 ... yes 106/20 ... yes 103/21 ... yes 108/22 ... yes 78/23 ... yes 109/24 ... yes 111/25 ... yes 112/26 ... yes 114/28 ... yes 114/29 ... yes 115/30 ... yes 116/31 ... yes 117/32 ... yes 117/33 ... yes 118/34 ... yes 116/35 ... yes 111/37 ... yes 120/40 ... yes 120/42 ... yes 120/43 ... yes 108/44 ... yes 90/45 ... yes 91/46 ... yes 129/112 ... yes 129/113 ... yes 129/114 ... yes 129/115 ... yes 129/116 ... yes 129/117 ... yes 129/118 ... yes 129/119 ... yes 129/120 ... yes 129/121 ... yes 129/122 ... yes 129/123 ... yes 129/124 ... yes 129/125 ... yes 129/126 ... yes 129/127 ... yes 129/128 ... yes 129/129 ... yes 129/130 ... yes 129/131 ... yes 129/132 ... yes 129/133 ... yes 129/134 ... yes 129/135 ... yes 129/136 ... yes 129/137 ... yes 129/138 ... yes 129/139 ... yes 129/140 ... yes 129/141 ... yes 129/142 ... yes 129/143 ... yes 129/144 ... yes 129/145 ... yes 129/146 ... yes 129/147 ... yes 129/148 ... yes 129/149 ... yes 129/150 ... yes 129/151 ... yes 129/152 ... yes 129/153 ... yes 129/154 ... yes 129/155 ... yes 129/156 ... yes 129/157 ... yes 129/158 ... yes 129/159 ... yes 129/160 ... yes 129/161 ... yes 129/162 ... yes 129/163 ... yes 106/165 ... yes 130/166 ... yes 94/167 ... yes 114/168 ... yes 130/169 ... yes 102/170 ... yes 108/172 ... yes 130/173 ... yes 105/175 ... yes 111/177 ... yes 133/178 ... yes 133/179 ... yes 124/180 ... yes 129/181 ... yes 91/182 ... yes 117/183 ... yes 141/184 ... yes 141/185 ... yes 145/186 ... yes 129/187 ... yes 146/188 ... yes 116/190 ... yes 147/191 ... yes 146/192 ... yes 2/194 ... yes Redis version >= 2.8.0? ... yes Ruby version >= 2.5.3 ? ... yes (2.6.3) Git version >= 2.21.0 ? ... yes (2.21.0) Git user has default SSH configuration? ... yes Active users: ... 84

Checking GitLab App ... Finished

Checking GitLab subtasks ... Finished

Edited Aug 06, 2019 by Pier-Luc Gagnon