Cannot migrate job artifacts to object storage
Summary
I have a GitLab CE install which has been upgraded progressively from 10.2 to currently 10.7.3. On upgrade to 10.6, I enabled object storage for CI job artifacts, using Google Cloud Storage as a backend. The gitlab:artifacts:check
and gitlab:artifacts:migrate
Rake tasks fail, and I'm not sure what to do about it.
Steps to reproduce
Don't have a good reproducer, I'm afraid.
Example Project
Also N/A. :(
What is the current bug behavior?
When I run gitlab:artifacts:check
, numerous artifacts are variously reported as having a missing checksum, no artifacts zip file, or no metadata.tgz.
When I run gitlab:artifacts:migrate
, I variously get 'request entity too large' or 'bad request' HTTP errors, or EPIPE
returned.
What is the expected correct behavior?
No check or migration failures for the artifacts.
Relevant logs and/or screenshots
These jobs are (mostly) available to view on https://gitlab.freedesktop.org. I can't find any particular pattern to them: job 19 which failed with 'request entity too large' fails when trying to upload a 53kB zip file. Job 239 which has no artifacts.zip was skipped before execution; job 240 which has no metadata.tgz ran to completion and is not obviously missing any metadata in the web UI.
Expand for verbose gitlab:artifacts:check output
root@gitlab-prod-gitlab-6cc797d8ff-gqmht:/# gitlab-rake gitlab:artifacts:check VERBOSE=true Checking integrity of Job artifacts - 1..208: Failures: 22 - Job artifact: 1: #<RuntimeError: Checksum missing> - Job artifact: 2: #<RuntimeError: Checksum missing> - Job artifact: 3: #<RuntimeError: Checksum missing> - Job artifact: 4: #<RuntimeError: Checksum missing> - Job artifact: 5: #<RuntimeError: Checksum missing> - Job artifact: 8: #<RuntimeError: Checksum missing> - Job artifact: 9: #<Errno::ENOENT: No such file or directory @ rb_sysopen - 45/23/4523540f1504cd17100c4835e85b7eefd49911580f8efff0599a8f283be6b9e3/2018_03_18/8/9/artifacts.zip> - Job artifact: 10: #<Errno::ENOENT: No such file or directory @ rb_sysopen - 45/23/4523540f1504cd17100c4835e85b7eefd49911580f8efff0599a8f283be6b9e3/2018_03_18/8/10/metadata.gz> - Job artifact: 11: #<RuntimeError: Checksum missing> - Job artifact: 14: #<RuntimeError: Checksum missing> - Job artifact: 17: #<RuntimeError: Checksum missing> - Job artifact: 20: #<RuntimeError: Checksum missing> - Job artifact: 21: #<RuntimeError: Checksum missing> - Job artifact: 22: #<RuntimeError: Checksum missing> - Job artifact: 23: #<RuntimeError: Checksum missing> - Job artifact: 26: #<RuntimeError: Checksum missing> - Job artifact: 28: #<Errno::ENOENT: No such file or directory @ rb_sysopen - 45/23/4523540f1504cd17100c4835e85b7eefd49911580f8efff0599a8f283be6b9e3/2018_04_10/21/28/artifacts.zip> - Job artifact: 29: #<RuntimeError: Checksum missing> - Job artifact: 127: #<Errno::ENOENT: No such file or directory @ rb_sysopen - 45/23/4523540f1504cd17100c4835e85b7eefd49911580f8efff0599a8f283be6b9e3/2018_04_20/123/127/artifacts.zip> - Job artifact: 128: #<Errno::ENOENT: No such file or directory @ rb_sysopen - 45/23/4523540f1504cd17100c4835e85b7eefd49911580f8efff0599a8f283be6b9e3/2018_04_20/123/128/metadata.gz> - Job artifact: 131: #<RuntimeError: Checksum missing> - Job artifact: 168: #<Errno::ENOENT: No such file or directory @ rb_sysopen - d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35/2018_04_30/161/168/metadata.gz> - 209..286: Failures: 5 - Job artifact: 242: #<Errno::ENOENT: No such file or directory @ rb_sysopen - 31/48/31489056e0916d59fe3add79e63f095af3ffb81604691f21cad442a85c7be617/2018_05_05/262/242/artifacts.zip> - Job artifact: 236: #<Errno::ENOENT: No such file or directory @ rb_sysopen - 31/48/31489056e0916d59fe3add79e63f095af3ffb81604691f21cad442a85c7be617/2018_05_05/259/236/metadata.gz> - Job artifact: 235: #<Errno::ENOENT: No such file or directory @ rb_sysopen - 31/48/31489056e0916d59fe3add79e63f095af3ffb81604691f21cad442a85c7be617/2018_05_05/259/235/artifacts.zip> - Job artifact: 239: #<Errno::ENOENT: No such file or directory @ rb_sysopen - 31/48/31489056e0916d59fe3add79e63f095af3ffb81604691f21cad442a85c7be617/2018_05_05/261/239/artifacts.zip> - Job artifact: 240: #<Errno::ENOENT: No such file or directory @ rb_sysopen - 31/48/31489056e0916d59fe3add79e63f095af3ffb81604691f21cad442a85c7be617/2018_05_05/261/240/metadata.gz> Done!
Expand for verbose gitlab:artifacts:migrate output
root@gitlab-prod-gitlab-6cc797d8ff-gqmht:/# gitlab-rake gitlab:artifacts:migrate VERBOSE=true I, [2018-05-07T12:02:49.992567 #674] INFO -- : Starting transfer of artifacts I, [2018-05-07T12:02:50.173304 #674] INFO -- : Transferred artifacts of 8 of 2852 to object storage E, [2018-05-07T12:02:50.356373 #674] ERROR -- : Failed to transfer artifacts of 19 with error: Expected(200) <=> Actual(413 Request Entity Too Large) excon.error.response :body => "[...]<p><b>413.</b> <ins>That\xE2\x80\x99s an error.</ins>\n <p>Your client issued a request that was too large.\n <script>\n [...]<ins>That\xE2\x80\x99s all we know.</ins>\n" :cookies => [ ] :headers => { "Content-Length" => "2398" "Content-Type" => "text/html; charset=UTF-8" "Date" => "Mon, 07 May 2018 12:02:50 GMT" "Referrer-Policy" => "no-referrer" } :path => "/d4%2F73%2Fd4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35%2F2018_04_10%2F19%2F25%2Fartifacts.zip" :port => 443 :reason_phrase => "Request Entity Too Large" :status => 413 :status_line => "HTTP/1.0 413 Request Entity Too Large\r\n"E, [2018-05-07T12:02:50.465572 #674] ERROR -- : Failed to transfer artifacts of 21 with error: Expected(200) <=> Actual(400 Bad Request) excon.error.response
:body => "[...] <p><b>400.</b> <ins>That\xE2\x80\x99s an error.</ins>\n <p>Your client has issued a malformed or illegal request. <ins>That\xE2\x80\x99s all we know.</ins>\n" :cookies => [ ] :headers => { "Content-Length" => "1555" "Content-Type" => "text/html; charset=UTF-8" "Date" => "Mon, 07 May 2018 12:02:50 GMT" "Server" => "UploadServer" } :path => "/45%2F23%2F4523540f1504cd17100c4835e85b7eefd49911580f8efff0599a8f283be6b9e3%2F2018_04_10%2F21%2F29%2Fmetadata.gz" :port => 443 :reason_phrase => "Bad Request" :status => 400 :status_line => "HTTP/1.1 400 Bad Request\r\n"I, [2018-05-07T12:02:50.473541 #674] INFO -- : Transferred artifacts of 123 of 2831 to object storage E, [2018-05-07T12:02:50.809551 #674] ERROR -- : Failed to transfer artifacts of 124 with error: Broken pipe (Errno::EPIPE) E, [2018-05-07T12:02:50.916282 #674] ERROR -- : Failed to transfer artifacts of 161 with error: Expected(200) <=> Actual(400 Bad Request) excon.error.response :body => "[...] <p><b>400.</b> <ins>That\xE2\x80\x99s an error.</ins>\n <p>Your client has issued a malformed or illegal request. <ins>That\xE2\x80\x99s all we know.</ins>\n" :cookies => [ ] :headers => { "Content-Length" => "1555" "Content-Type" => "text/html; charset=UTF-8" "Date" => "Mon, 07 May 2018 12:02:50 GMT" "Server" => "UploadServer" } :path => "/d4%2F73%2Fd4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35%2F2018_04_30%2F161%2F167%2Fartifacts.zip" :port => 443 :reason_phrase => "Bad Request" :status => 400 :status_line => "HTTP/1.1 400 Bad Request\r\n"
E, [2018-05-07T12:02:51.344981 #674] ERROR -- : Failed to transfer artifacts of 163 with error: Broken pipe (Errno::EPIPE) I, [2018-05-07T12:02:51.353893 #674] INFO -- : Transferred artifacts of 259 of 37438 to object storage I, [2018-05-07T12:02:51.359944 #674] INFO -- : Transferred artifacts of 261 of 41320 to object storage E, [2018-05-07T12:02:51.462130 #674] ERROR -- : Failed to transfer artifacts of 262 with error: Expected(200) <=> Actual(400 Bad Request) excon.error.response :body => "[...] <p><b>400.</b> <ins>That\xE2\x80\x99s an error.</ins>\n <p>Your client has issued a malformed or illegal request. <ins>That\xE2\x80\x99s all we know.</ins>\n" :cookies => [ ] :headers => { "Content-Length" => "1555" "Content-Type" => "text/html; charset=UTF-8" "Date" => "Mon, 07 May 2018 12:02:51 GMT" "Server" => "UploadServer" } :path => "/31%2F48%2F31489056e0916d59fe3add79e63f095af3ffb81604691f21cad442a85c7be617%2F2018_05_05%2F262%2F243%2Fmetadata.gz" :port => 443 :reason_phrase => "Bad Request" :status => 400 :status_line => "HTTP/1.1 400 Bad Request\r\n"
E, [2018-05-07T12:02:51.577655 #674] ERROR -- : Failed to transfer artifacts of 264 with error: Expected(200) <=> Actual(400 Bad Request) excon.error.response :body => "[...] <p><b>400.</b> <ins>That\xE2\x80\x99s an error.</ins>\n <p>Your client has issued a malformed or illegal request. <ins>That\xE2\x80\x99s all we know.</ins>\n" :cookies => [ ] :headers => { "Content-Length" => "1555" "Content-Type" => "text/html; charset=UTF-8" "Date" => "Mon, 07 May 2018 12:02:51 GMT" "Server" => "UploadServer" } :path => "/d4%2F73%2Fd4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35%2F2018_05_05%2F264%2F246%2Fartifacts.zip" :port => 443 :reason_phrase => "Bad Request" :status => 400 :status_line => "HTTP/1.1 400 Bad Request\r\n"
E, [2018-05-07T12:02:51.806013 #674] ERROR -- : Failed to transfer artifacts of 266 with error: Broken pipe (Errno::EPIPE)
Output of checks
Results of GitLab environment info
This is the 10.7.3 gitlab-ce Omnibus Docker container, running on GKE.
Expand for output related to GitLab environment info
System information System: Current User: git Using RVM: no Ruby Version: 2.3.6p384 Gem Version: 2.6.13 Bundler Version:1.13.7 Rake Version: 12.3.0 Redis Version: 3.2.11 Git Version: 2.14.3 Sidekiq Version:5.0.5 Go Version: unknown
GitLab information Version: 10.7.3 Revision: 2555d6c Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: postgresql URL: https://gitlab.freedesktop.org HTTP Clone URL: https://gitlab.freedesktop.org/some-group/some-project.git SSH Clone URL: git@gitlab.freedesktop.org:some-group/some-project.git Using LDAP: no Using Omniauth: yes Omniauth Providers: google_oauth2, gitlab, github, twitter
GitLab Shell Version: 7.1.2 Repository storage paths:
- default: /gitlab-data/git-data/repositories Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks Git: /opt/gitlab/embedded/bin/git
Expand for artifacts configuration section of Omnibus gitlab.rb
gitlab_rails['artifacts_enabled'] = true gitlab_rails['artifacts_object_store_enabled'] = true gitlab_rails['artifacts_object_store_remote_directory'] = '[elided]' # XXX: When we move to 10.7, we can use JSON keys rather than old interop gitlab_rails['artifacts_object_store_connection'] = { 'provider' => 'Google', 'google_storage_access_key_id' => ENV['GOOGLE_ARTIFACT_ACCESS_KEY'], 'google_storage_secret_access_key' => ENV['GOOGLE_ARTIFACT_SECRET_KEY'], }
Results of GitLab application Check
All OK.
Expand for output related to the GitLab application check
Checking GitLab Shell ...GitLab Shell version >= 7.1.2 ? ... OK (7.1.2) Repo base directory exists? default... yes Repo storage directories are symlinks? default... no Repo paths owned by git:root, or git:git? default... yes Repo paths access is drwxrws---? default... yes hooks directories in repos are links: ... 1222/1 ... repository is empty 1222/2 ... ok 1222/3 ... ok 1222/4 ... ok 1332/5 ... ok 1345/6 ... ok 1345/7 ... ok 1345/10 ... ok 1345/11 ... ok 255/14 ... ok 255/15 ... ok 255/16 ... ok 1348/17 ... ok 1345/18 ... ok 1345/19 ... ok 1020/20 ... ok 1020/21 ... ok 1020/22 ... ok 1020/23 ... ok 1099/24 ... ok 1308/25 ... ok 1336/26 ... ok 255/27 ... repository is empty 255/28 ... ok 1099/30 ... ok 255/31 ... ok 1357/32 ... ok 1349/34 ... repository is empty 1200/35 ... ok 123/36 ... ok 1150/37 ... ok 1150/38 ... ok 1305/40 ... ok 736/41 ... ok 1385/42 ... ok 1098/43 ... ok 123/44 ... ok 1098/45 ... ok 1033/46 ... ok 1353/47 ... ok 1403/48 ... ok 1099/49 ... ok 1098/51 ... ok 1124/52 ... ok 1152/53 ... repository is empty 1305/54 ... ok 1305/55 ... ok 1305/56 ... ok 1290/57 ... ok 1199/58 ... ok 1406/59 ... ok 1199/60 ... ok 1199/61 ... ok 1199/62 ... ok 1199/63 ... ok 1199/64 ... ok 1199/65 ... ok 1199/66 ... ok 1199/67 ... ok 1199/68 ... ok 1199/69 ... repository is empty 1199/70 ... ok 1199/71 ... repository is empty 1199/72 ... ok 1199/73 ... ok 1407/74 ... ok 1407/75 ... ok 1407/76 ... ok 1407/77 ... ok 1407/78 ... ok 1407/79 ... ok Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Check GitLab API access: OK Redis available via internal API: OK
Access to /gitlab-data/ssh/authorized_keys: OK gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Sidekiq ...
Running? ... yes Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Reply by email is disabled in config/gitlab.yml Checking LDAP ...
LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab ...
Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ... 1222/1 ... yes 1222/2 ... yes 1222/3 ... yes 1222/4 ... yes 1332/5 ... yes 1345/6 ... yes 1345/7 ... yes 1345/10 ... yes 1345/11 ... yes 255/14 ... yes 255/15 ... yes 255/16 ... yes 1348/17 ... yes 1345/18 ... yes 1345/19 ... yes 1020/20 ... yes 1020/21 ... yes 1020/22 ... yes 1020/23 ... yes 1099/24 ... yes 1308/25 ... yes 1336/26 ... yes 255/27 ... yes 255/28 ... yes 1099/30 ... yes 255/31 ... yes 1357/32 ... yes 1349/34 ... yes 1200/35 ... yes 123/36 ... yes 1150/37 ... yes 1150/38 ... yes 1305/40 ... yes 736/41 ... yes 1385/42 ... yes 1098/43 ... yes 123/44 ... yes 1098/45 ... yes 1033/46 ... yes 1353/47 ... yes 1403/48 ... yes 1099/49 ... yes 1098/51 ... yes 1124/52 ... yes 1152/53 ... yes 1305/54 ... yes 1305/55 ... yes 1305/56 ... yes 1290/57 ... yes 1199/58 ... yes 1406/59 ... yes 1199/60 ... yes 1199/61 ... yes 1199/62 ... yes 1199/63 ... yes 1199/64 ... yes 1199/65 ... yes 1199/66 ... yes 1199/67 ... yes 1199/68 ... yes 1199/69 ... yes 1199/70 ... yes 1199/71 ... yes 1199/72 ... yes 1199/73 ... yes 1407/74 ... yes 1407/75 ... yes 1407/76 ... yes 1407/77 ... yes 1407/78 ... yes 1407/79 ... yes Redis version >= 2.8.0? ... yes Ruby version >= 2.3.5 ? ... yes (2.3.6) Git version >= 2.9.5 ? ... yes (2.14.3) Git user has default SSH configuration? ... yes Active users: ... 1189
Checking GitLab ... Finished
Possible fixes
Is there anything I can do about this, or any way I can help investigate further? If the 'fix' is just waiting for the job artifacts to expire out then that's doable. It would be nice to be able to migrate all the artifacts out though, so I have one less thing to worry about.