Setting artifacts background upload makes download fail
Summary
When the artifacts_object_store_background_upload is enable, artifacts and logs download fails (500 Error).
In our case, we have openstack/swift configured as backend provider. This provider does only support background uploads.
Steps to reproduce
- Set config:
gitlab_rails['artifacts_enabled'] = true
gitlab_rails['artifacts_object_store_background_upload'] = true
gitlab_rails['artifacts_object_store_connection'] = {"openstack_api_key"=>"xxx", "openstack_auth_url"=>"xxx", "openstack_tenant_id"=>"xxx", "openstack_username"=>"xxx", "provider"=>"openstack"}
gitlab_rails['artifacts_object_store_direct_upload'] = false
gitlab_rails['artifacts_object_store_enabled'] = true
gitlab_rails['artifacts_object_store_proxy_download'] = false
gitlab_rails['artifacts_object_store_remote_directory'] = "artifacts"
2. Run a job (with or without artifacts, as also the job trace will be stored as artifact). 3. When the build is done refresh the page or try to download the generated artifacts.
Example Project
n/a
What is the current bug behavior?
When above settings are applied, the data seems to get upload to the objectstore. I do see data appearing in the objectstore and the logfile seems to correspond with that of the run. Retrieving any of the artifact data through gitlab will result in a 500 error.
I highly doubt credentials are the issue, as uploading to the objectstore is no problem.
Note: tried with setting artifacts_object_store_proxy_download to true (although we don't use any proxy), but this doesn't help.
What is the expected correct behavior?
(What you should see instead)
Relevant logs and/or screenshots
I only see relevant (yet very very little verbose) data in the rails production log:
Started GET "/devops/puppet/-/jobs/232795/artifacts/browse" for 127.0.0.1 at 2019-02-14 11:40:32 +0100
Processing by Gitlab::RequestForgeryProtection::Controller#index as HTML
Completed 200 OK in 0ms (ActiveRecord: 0.0ms)
Processing by Projects::ArtifactsController#browse as HTML
Parameters: {"namespace_id"=>"devops", "project_id"=>"puppet", "job_id"=>"232795"}
Completed 500 Internal Server Error in 114ms (ActiveRecord: 16.2ms)
Gitlab::HttpIO::InvalidURLError (Gitlab::HttpIO::InvalidURLError):
lib/gitlab/http_io.rb:20:in `initialize'
app/uploaders/gitlab_uploader.rb:89:in `new'
app/uploaders/gitlab_uploader.rb:89:in `open'
app/models/ci/build.rb:573:in `artifacts_metadata_entry'
app/controllers/projects/artifacts_controller.rb:24:in `browse'
lib/gitlab/i18n.rb:55:in `with_locale'
lib/gitlab/i18n.rb:61:in `with_user_locale'
app/controllers/application_controller.rb:417:in `set_locale'
lib/gitlab/middleware/rails_queue_duration.rb:24:in `call'
lib/gitlab/metrics/rack_middleware.rb:17:in `block in call'
lib/gitlab/metrics/transaction.rb:55:in `run'
lib/gitlab/metrics/rack_middleware.rb:17:in `call'
lib/gitlab/middleware/multipart.rb:103:in `call'
lib/gitlab/request_profiler/middleware.rb:16:in `call'
lib/gitlab/middleware/go.rb:20:in `call'
lib/gitlab/etag_caching/middleware.rb:13:in `call'
lib/gitlab/middleware/correlation_id.rb:16:in `block in call'
lib/gitlab/correlation_id.rb:15:in `use_id'
lib/gitlab/middleware/correlation_id.rb:15:in `call'
lib/gitlab/middleware/read_only/controller.rb:40:in `call'
lib/gitlab/middleware/read_only.rb:18:in `call'
lib/gitlab/middleware/basic_health_check.rb:25:in `call'
lib/gitlab/request_context.rb:20:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:29:in `call'
lib/gitlab/middleware/release_env.rb:13:in `call'
The request for the trace log gives a similar error.
Related data in openstack:
Expand for objectstore contents listing
$VAR1 = [ { 'bytes' => 31187842, 'hash' => 'xxx', 'name' => '5a/39/5a39cadd1b007093db50744797c7a04a34f73b35ed444704206705b02597d6fd/2019_02_13/232795/235176/devops_770_server.zip', 'content_type' => 'application/zip', 'last_modified' => '2019-02-13T22:53:27.629440' }, { 'name' => '5a/39/5a39cadd1b007093db50744797c7a04a34f73b35ed444704206705b02597d6fd/2019_02_13/232795/235177/metadata.gz', 'hash' => 'xxx', 'bytes' => 159275, 'content_type' => 'application/gzip', 'last_modified' => '2019-02-13T22:53:27.581380' }, { 'last_modified' => '2019-02-13T22:53:28.727570', 'content_type' => 'text/plain', 'hash' => 'xxx', 'bytes' => 210931, 'name' => '5a/39/5a39cadd1b007093db50744797c7a04a34f73b35ed444704206705b02597d6fd/2019_02_13/232795/235178/job.log' } ];
Related data in database:
Expand for select * from ci_job_artifacts where job_id=232795;
id | project_id | job_id | file_type | size | created_at | updated_at | expire_at | file | file_sha256 | file_store | file_format | file_loca tion --------+------------+--------+-----------+----------+-------------------------------+-------------------------------+-------------------------------+---------------------- -+------------------------------------------------------------------------------------------------------------------------------------+------------+-------------+---------- ----- 235176 | 199 | 232795 | 1 | 31187842 | 2019-02-13 22:53:24.967539+00 | 2019-02-13 22:53:24.967539+00 | 2019-03-15 22:53:24.952022+00 | devops_770_server.zip | \x...xxx | 2 | 2 |235177 | 199 | 232795 | 2 | 159275 | 2019-02-13 22:53:24.976212+00 | 2019-02-13 22:53:24.976212+00 | 2019-03-15 22:53:24.955456+00 | metadata.gz | \x...xxx | 2 | 3 |
235178 | 199 | 232795 | 3 | 210931 | 2019-02-13 22:53:28.606636+00 | 2019-02-13 22:53:28.606636+00 | | job.log | \x...xxx | 2 | |
(3 rows)
Output of checks
n/a
Results of GitLab environment info
Expand for output related to GitLab environment info
System information System: Debian 9.7 Current User: git Using RVM: no Ruby Version: 2.5.3p105 Gem Version: 2.7.6 Bundler Version:1.16.6 Rake Version: 12.3.2 Redis Version: 3.2.12 Git Version: 2.18.1 Sidekiq Version:5.2.3 Go Version: unknownGitLab information Version: 11.7.5 Revision: c5b5b18 Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: postgresql URL: https://test-gitlab.xxx.nl HTTP Clone URL: https://test-gitlab.xxx.nl/some-group/some-project.git SSH Clone URL: git@test-gitlab.xxx.nl:some-group/some-project.git Using LDAP: no Using Omniauth: yes Omniauth Providers: saml
Results of GitLab application Check
Expand for output related to the GitLab application check
Checking GitLab subtasks ...Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 8.4.4 ? ... OK (8.4.4) Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Check GitLab API access: OK Redis available via internal API: OK
Access to /var/opt/gitlab/.ssh/authorized_keys: OK gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab App ...
Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ... 4/1 ... yes 3/6 ... yes 7/7 ... yes 4/8 ... yes 4/9 ... yes 4/10 ... yes 11/11 ... yes 4/12 ... yes 4/14 ... yes 7/15 ... yes 4/16 ... yes 4/17 ... yes 4/18 ... yes 4/20 ... yes 6/21 ... yes 4/22 ... yes 4/23 ... yes 4/24 ... yes 34/26 ... yes 4/27 ... yes 4/28 ... yes 4/29 ... yes 4/30 ... yes 34/31 ... yes 4/32 ... yes 4/35 ... yes 4/37 ... yes 4/38 ... yes 4/39 ... yes 4/40 ... yes 4/41 ... yes 4/42 ... yes 4/44 ... yes 4/45 ... yes 4/46 ... yes 4/47 ... yes 4/48 ... yes 4/49 ... yes 4/50 ... yes 4/51 ... yes 4/52 ... yes 44/53 ... yes 4/54 ... yes 44/55 ... yes 4/57 ... yes 4/58 ... yes 4/60 ... yes 4/61 ... yes 4/62 ... yes 4/63 ... yes 4/65 ... yes 4/66 ... yes 4/67 ... yes 4/68 ... yes 4/69 ... yes 3/70 ... yes 4/71 ... yes 4/72 ... yes 4/73 ... yes 4/74 ... yes 4/75 ... yes 4/76 ... yes 4/77 ... yes 4/78 ... yes 4/79 ... yes 4/80 ... yes 4/81 ... yes 4/82 ... yes 4/83 ... yes 4/84 ... yes 4/85 ... yes 4/86 ... yes 4/87 ... yes 4/88 ... yes 4/89 ... yes 4/90 ... yes 13/91 ... yes 13/92 ... yes 13/93 ... yes 4/94 ... yes 6/95 ... yes 4/96 ... yes 52/97 ... yes 52/98 ... yes 52/99 ... yes 52/100 ... yes 52/102 ... yes 52/103 ... yes 52/104 ... yes 52/105 ... yes 52/106 ... yes 52/107 ... yes 52/108 ... yes 52/109 ... yes 52/110 ... yes 52/111 ... yes 52/112 ... yes 52/113 ... yes 52/114 ... yes 44/115 ... yes 44/116 ... yes 59/117 ... yes 44/118 ... yes 4/119 ... yes 52/120 ... yes 59/121 ... yes 44/122 ... yes 44/123 ... yes 56/124 ... yes 52/125 ... yes 52/127 ... yes 13/129 ... yes 4/130 ... yes 4/131 ... yes 4/132 ... yes 11/133 ... yes 56/134 ... yes 4/135 ... yes 67/137 ... yes 67/138 ... yes 4/139 ... yes 56/140 ... yes 4/141 ... yes 44/142 ... yes 4/143 ... yes 44/146 ... yes 44/147 ... yes 44/148 ... yes 56/149 ... yes 79/150 ... yes 79/151 ... yes 44/152 ... yes 44/153 ... yes 4/154 ... yes 4/155 ... yes 79/156 ... yes 4/157 ... yes 4/158 ... yes 4/159 ... yes 52/161 ... yes 25/162 ... yes 86/164 ... yes 73/166 ... yes 86/167 ... yes 86/168 ... yes 86/169 ... yes 86/170 ... yes 86/171 ... yes 86/172 ... yes 4/173 ... yes 25/174 ... yes 25/175 ... yes 52/176 ... yes 87/177 ... yes 4/178 ... yes 38/179 ... yes 4/180 ... yes 86/181 ... yes 86/182 ... yes 4/183 ... yes 44/185 ... yes 52/186 ... yes 79/188 ... yes 52/189 ... yes 79/190 ... yes 92/191 ... yes 52/192 ... yes 52/193 ... yes 79/194 ... yes 98/195 ... yes 132/196 ... yes 70/197 ... yes 52/198 ... yes 79/199 ... yes 92/200 ... yes 4/201 ... yes 44/202 ... yes 103/203 ... yes 103/204 ... yes 103/205 ... yes 103/206 ... yes 44/209 ... yes 79/210 ... yes 52/211 ... yes 94/212 ... yes 78/213 ... yes 52/214 ... yes 79/215 ... yes 92/216 ... yes 4/219 ... yes 105/221 ... yes 4/223 ... yes 25/224 ... yes 64/225 ... yes 56/226 ... yes 77/227 ... yes 132/228 ... yes 4/231 ... yes 132/232 ... yes 56/233 ... yes 79/234 ... yes 4/235 ... yes 52/236 ... yes 52/237 ... yes 7/238 ... yes 79/239 ... yes 63/240 ... yes 94/241 ... yes 2/243 ... yes 79/244 ... yes 79/245 ... yes 79/246 ... yes 7/247 ... yes 117/248 ... yes 117/249 ... yes 117/250 ... yes 117/251 ... yes 117/252 ... yes 117/253 ... yes 117/254 ... yes 44/255 ... yes 104/256 ... yes 52/257 ... yes 3/258 ... yes 3/259 ... yes 3/260 ... yes 3/261 ... yes 79/262 ... yes 92/263 ... yes 92/264 ... yes 52/265 ... yes 78/267 ... yes 119/268 ... yes 119/269 ... yes 79/270 ... yes 4/271 ... yes 121/272 ... yes 4/273 ... yes 52/274 ... yes 4/275 ... yes 4/276 ... yes 25/277 ... yes 79/278 ... yes 124/279 ... yes 4/280 ... yes 4/281 ... yes 4/282 ... yes 92/283 ... yes 92/284 ... yes 4/285 ... yes 79/286 ... yes 4/288 ... yes 4/289 ... yes 65/290 ... yes 52/291 ... yes 129/292 ... yes 129/293 ... yes 129/294 ... yes 129/295 ... yes 129/296 ... yes 129/297 ... yes 129/298 ... yes 129/299 ... yes 129/300 ... yes 129/301 ... yes 129/302 ... yes 129/303 ... yes 129/304 ... yes 129/305 ... yes 129/306 ... yes 129/307 ... yes 129/308 ... yes 129/309 ... yes 129/310 ... yes 129/311 ... yes 129/312 ... yes 129/313 ... yes 129/314 ... yes 129/315 ... yes 129/316 ... yes 129/317 ... yes 129/318 ... yes 129/319 ... yes 129/320 ... yes 129/321 ... yes 129/322 ... yes 129/323 ... yes 129/324 ... yes 129/325 ... yes 129/326 ... yes 129/327 ... yes 129/328 ... yes 129/329 ... yes 129/330 ... yes 129/331 ... yes 129/332 ... yes 129/333 ... yes 129/334 ... yes 129/335 ... yes 129/336 ... yes 129/337 ... yes 129/338 ... yes 129/339 ... yes 129/340 ... yes 129/341 ... yes 129/342 ... yes 129/343 ... yes 201/344 ... yes 79/345 ... yes 52/347 ... yes 128/348 ... yes 79/351 ... yes 79/352 ... yes 13/353 ... yes 52/355 ... yes 88/356 ... yes 4/357 ... yes 60/358 ... yes 60/359 ... yes 4/360 ... yes 132/361 ... yes 141/362 ... yes 141/363 ... yes 141/364 ... yes 94/365 ... yes 88/366 ... yes 13/367 ... yes 88/368 ... yes 4/369 ... yes 146/373 ... yes 4/374 ... yes 79/375 ... yes 13/376 ... yes 4/378 ... yes 79/380 ... yes 4/381 ... yes 56/382 ... yes 15/383 ... yes 79/385 ... yes 94/386 ... yes 149/387 ... yes 2/388 ... yes 146/389 ... yes 44/390 ... yes 92/391 ... yes 44/392 ... yes 52/393 ... yes 92/394 ... yes 52/395 ... yes 52/396 ... yes 4/397 ... yes 105/398 ... yes 52/399 ... yes 4/400 ... yes 125/401 ... yes 78/402 ... yes 4/403 ... yes 4/404 ... yes 4/405 ... yes 44/406 ... yes 52/407 ... yes 146/408 ... yes 157/409 ... yes 52/410 ... yes 94/411 ... yes 160/413 ... yes 160/414 ... yes 160/418 ... yes 160/419 ... yes 159/420 ... yes 159/421 ... yes 25/422 ... yes 25/423 ... yes 146/424 ... yes 146/425 ... yes 168/427 ... yes 44/428 ... yes 149/429 ... yes 2/430 ... yes 167/431 ... yes 149/432 ... yes 4/433 ... yes 4/434 ... yes 63/436 ... yes 159/437 ... yes 146/438 ... yes 25/439 ... yes 146/441 ... yes 157/442 ... yes 105/444 ... yes 167/445 ... yes 167/446 ... yes 167/447 ... yes 167/448 ... yes 157/451 ... yes 146/452 ... yes 105/453 ... yes 146/454 ... yes 105/455 ... yes 88/457 ... yes 146/458 ... yes 4/459 ... yes 146/460 ... yes 171/461 ... yes 171/462 ... yes 157/463 ... yes 52/464 ... yes 160/465 ... yes 160/467 ... yes 78/468 ... yes 160/469 ... yes 152/470 ... yes 92/471 ... yes 146/473 ... yes 160/474 ... yes 160/475 ... yes 160/476 ... yes 146/477 ... yes 160/478 ... yes 157/479 ... yes 38/480 ... yes 44/481 ... yes 92/482 ... yes 146/483 ... yes 146/484 ... yes 160/485 ... yes 171/486 ... yes 160/487 ... yes 17/488 ... yes 4/489 ... yes 4/490 ... yes 167/491 ... yes 167/493 ... yes 167/494 ... yes 160/495 ... yes 4/497 ... yes 160/498 ... yes 4/499 ... yes 146/500 ... yes 160/501 ... yes 167/505 ... yes 160/506 ... yes 38/507 ... yes 3/508 ... yes 146/510 ... yes 89/512 ... yes 167/513 ... yes 38/516 ... yes 105/517 ... yes 132/518 ... yes 146/519 ... yes 160/520 ... yes 105/521 ... yes 157/523 ... yes 157/528 ... yes 4/529 ... yes 17/530 ... yes 167/531 ... yes 160/532 ... yes 167/533 ... yes 96/534 ... yes 17/536 ... yes 167/537 ... yes 78/538 ... yes 149/539 ... yes 4/540 ... yes 60/541 ... yes 60/542 ... yes 157/543 ... yes 157/544 ... yes 52/545 ... yes 146/546 ... yes 60/547 ... yes 157/548 ... yes 157/549 ... yes 157/550 ... yes 182/551 ... yes 178/555 ... yes 160/556 ... yes 182/557 ... yes 149/558 ... yes 157/559 ... yes 17/560 ... yes 52/561 ... yes 79/563 ... yes 146/564 ... yes 3/565 ... yes 89/566 ... yes 182/567 ... yes 183/570 ... yes 183/571 ... yes 6/574 ... yes 183/575 ... yes 167/576 ... yes 167/577 ... yes 184/578 ... yes 185/579 ... yes 182/580 ... yes 146/581 ... yes 49/582 ... yes 16/584 ... yes 157/585 ... yes 38/588 ... yes 4/589 ... yes 157/590 ... yes 157/591 ... yes 16/592 ... yes 16/593 ... yes 167/594 ... yes 52/595 ... yes 4/596 ... yes 2/597 ... yes 49/598 ... yes 160/599 ... yes 4/601 ... yes 52/602 ... yes 44/603 ... yes 185/604 ... yes 2/605 ... yes 4/606 ... yes 146/607 ... yes 43/608 ... yes 185/609 ... yes 3/610 ... yes 183/612 ... yes 185/615 ... yes 157/616 ... yes 157/617 ... yes 185/618 ... yes 52/619 ... yes 79/620 ... yes 157/621 ... yes 3/622 ... yes 2/623 ... yes 157/624 ... yes 3/625 ... yes 3/629 ... yes 52/630 ... yes 49/631 ... yes 49/632 ... yes 25/633 ... yes 43/634 ... yes 3/635 ... yes 157/636 ... yes 146/637 ... yes 146/638 ... yes 146/639 ... yes 167/640 ... yes 16/641 ... yes 49/642 ... yes 4/643 ... yes 52/644 ... yes 16/645 ... yes 38/646 ... yes 52/647 ... yes 167/648 ... yes 146/649 ... yes 146/650 ... yes 16/651 ... yes 146/652 ... yes 146/653 ... yes 146/654 ... yes 157/655 ... yes 146/656 ... yes 38/657 ... yes 146/658 ... yes 185/659 ... yes 132/660 ... yes 43/661 ... yes 43/662 ... yes 157/663 ... yes 146/664 ... yes 157/665 ... yes 16/666 ... yes 146/667 ... yes 146/668 ... yes 146/669 ... yes 157/670 ... yes 157/671 ... yes 79/672 ... yes 168/673 ... yes 4/674 ... yes 185/676 ... yes 52/677 ... yes 52/678 ... yes 16/680 ... yes 169/681 ... yes 168/682 ... yes 146/683 ... yes 157/684 ... yes 91/685 ... yes 52/686 ... yes 146/688 ... yes 146/689 ... yes 53/691 ... yes 111/692 ... yes 52/693 ... yes 43/694 ... yes 72/695 ... yes 111/696 ... yes 201/697 ... yes 44/698 ... yes 146/699 ... yes 53/700 ... yes 111/701 ... yes 52/702 ... yes 52/703 ... yes 202/704 ... yes 52/705 ... yes 52/706 ... yes 146/708 ... yes 146/709 ... yes 132/710 ... yes 52/711 ... yes 38/712 ... yes 157/713 ... yes 157/714 ... yes 157/715 ... yes 4/716 ... yes 157/717 ... yes 157/718 ... yes 157/719 ... yes 146/721 ... yes 38/722 ... yes 38/723 ... yes 185/724 ... yes 4/726 ... yes 157/727 ... yes 157/728 ... yes 146/729 ... yes 38/730 ... yes 215/731 ... yes 215/732 ... yes 210/733 ... yes 215/734 ... yes 157/735 ... yes 157/736 ... yes 43/738 ... yes 216/739 ... yes 72/740 ... yes 52/741 ... yes 52/742 ... yes 216/743 ... yes 216/744 ... yes 216/745 ... yes 51/746 ... yes 157/747 ... yes 157/748 ... yes 157/749 ... yes 157/751 ... yes 157/752 ... yes 216/753 ... yes 146/754 ... yes 52/756 ... yes 43/757 ... yes 220/759 ... yes 157/760 ... yes 16/761 ... yes 157/762 ... yes 96/763 ... yes Redis version >= 2.8.0? ... yes Ruby version >= 2.3.5 ? ... yes (2.5.3) Git version >= 2.18.0 ? ... yes (2.18.1) Git user has default SSH configuration? ... yes Active users: ... 117
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished
Possible fixes
(If you can, link to the line of code that might be responsible for the problem)
I tried to be as verbose as possible, please ask if you are missing fragments or if I need to test stuff. Im happy to help where possible.
labels: Object Storage ~bug ~artifacts