Skip to content

can not use aliyun object storage for job artifacts

Summary

use aliyun oss object storage for job artifacts, when i migrate artifacts, it can not work

Steps to reproduce

you should have an aliyun oss bucket, and right access policy setup, you need the access key id and secret.

gitlab.rb

gitlab_rails['artifacts_object_store_enabled'] = true
gitlab_rails['artifacts_object_store_remote_directory'] = "example-bucket-name"
gitlab_rails['artifacts_object_store_connection'] = {
  'provider' => 'aliyun',
  'aliyun_accesskey_id' => '*********',
  'aliyun_accesskey_secret' => '**********',
  'aliyun_oss_bucket' => 'example-bucket-name',
  'aliyun_oss_location' => 'cn-shanghai',
  'aliyun_oss_endpoint' => 'oss-cn-shanghai.aliyuncs.com'
}

then run follow command

$ sudo gitlab-ctl reconfigure
$ sudo gitlab-rake gitlab:artifacts:migrate

What is the current bug behavior?

it return error logs

I, [2018-10-25T02:33:53.885875 #8127]  INFO -- : Starting transfer of artifacts
E, [2018-10-25T02:33:54.027448 #8127] ERROR -- : Failed to transfer artifacts of 1 with error: no implicit conversion of nil into String

What is the expected correct behavior?

it should exit with success, and migrate local artifacts to object storage

Relevant logs

==> /var/log/gitlab/sidekiq/current <==
2018-10-25_02:33:41.48735 2018-10-25T02:33:41.487Z 7886 TID-otl43zj7u ObjectStorage::BackgroundMoveWorker JID-52b56092c457b8d21b14afa8 INFO: start
2018-10-25_02:33:41.58575 2018-10-25T02:33:41.585Z 7886 TID-otl43zj7u ObjectStorage::BackgroundMoveWorker JID-52b56092c457b8d21b14afa8 INFO: Adding dead ObjectStorage::BackgroundMoveWorker job 52b56092c457b8d21b14afa8
2018-10-25_02:33:41.58645 2018-10-25T02:33:41.586Z 7886 TID-otl43zj7u ObjectStorage::BackgroundMoveWorker JID-52b56092c457b8d21b14afa8 INFO: fail: 0.099 sec
2018-10-25_02:33:41.58657 2018-10-25T02:33:41.586Z 7886 TID-otl43zj7u WARN: {"context":"Job raised exception","job":{"class":"ObjectStorage::BackgroundMoveWorker","args":["JobArtifactUploader","Ci::JobArtifact","file",2],"retry":5,"queue":"object_storage:object_storage_background_move","queue_namespace":"object_storage","jid":"52b56092c457b8d21b14afa8","created_at":1540434104.7881083,"enqueued_at":1540434821.4865055,"error_message":"no implicit conversion of nil into String","error_class":"TypeError","failed_at":1540434104.9099479,"retry_count":4,"retried_at":1540434467.7912023},"jobstr":"{\"class\":\"ObjectStorage::BackgroundMoveWorker\",\"args\":[\"JobArtifactUploader\",\"Ci::JobArtifact\",\"file\",2],\"retry\":5,\"queue\":\"object_storage:object_storage_background_move\",\"queue_namespace\":\"object_storage\",\"jid\":\"52b56092c457b8d21b14afa8\",\"created_at\":1540434104.7881083,\"enqueued_at\":1540434821.4865055,\"error_message\":\"no implicit conversion of nil into String\",\"error_class\":\"TypeError\",\"failed_at\":1540434104.9099479,\"retry_count\":4,\"retried_at\":1540434467.7912023}"}
2018-10-25_02:33:41.58663 2018-10-25T02:33:41.586Z 7886 TID-otl43zj7u WARN: TypeError: no implicit conversion of nil into String
2018-10-25_02:33:41.58671 2018-10-25T02:33:41.586Z 7886 TID-otl43zj7u WARN: /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/fog-aliyun-0.2.0/lib/fog/aliyun/storage.rb:113:in `+'
2018-10-25_02:33:41.58671 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/fog-aliyun-0.2.0/lib/fog/aliyun/storage.rb:113:in `request'
2018-10-25_02:33:41.58672 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/fog-aliyun-0.2.0/lib/fog/aliyun/requests/storage/get_bucket.rb:23:in `get_bucket_location'
2018-10-25_02:33:41.58672 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/fog-aliyun-0.2.0/lib/fog/aliyun/requests/storage/put_object.rb:13:in `put_object'
2018-10-25_02:33:41.58673 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/fog-aliyun-0.2.0/lib/fog/aliyun/models/storage/file.rb:113:in `save'
2018-10-25_02:33:41.58673 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/fog-core-1.45.0/lib/fog/core/collection.rb:50:in `create'
2018-10-25_02:33:41.58674 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/carrierwave-1.2.3/lib/carrierwave/storage/fog.rb:325:in `store'
2018-10-25_02:33:41.58674 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/carrierwave-1.2.3/lib/carrierwave/storage/fog.rb:78:in `store!'
2018-10-25_02:33:41.58674 /opt/gitlab/embedded/service/gitlab-rails/app/uploaders/object_storage.rb:430:in `block (2 levels) in unsafe_migrate!'
2018-10-25_02:33:41.58675 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/carrierwave-1.2.3/lib/carrierwave/uploader/callbacks.rb:15:in `with_callbacks'
2018-10-25_02:33:41.58675 /opt/gitlab/embedded/service/gitlab-rails/app/uploaders/object_storage.rb:429:in `block in unsafe_migrate!'
2018-10-25_02:33:41.58675 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/carrierwave-1.2.3/lib/carrierwave/uploader/callbacks.rb:15:in `with_callbacks'
2018-10-25_02:33:41.58676 /opt/gitlab/embedded/service/gitlab-rails/app/uploaders/object_storage.rb:428:in `unsafe_migrate!'
2018-10-25_02:33:41.58676 /opt/gitlab/embedded/service/gitlab-rails/app/uploaders/object_storage.rb:262:in `block in migrate!'
2018-10-25_02:33:41.58676 /opt/gitlab/embedded/service/gitlab-rails/app/uploaders/object_storage.rb:407:in `with_exclusive_lease'
2018-10-25_02:33:41.58676 /opt/gitlab/embedded/service/gitlab-rails/app/uploaders/object_storage.rb:261:in `migrate!'
2018-10-25_02:33:41.58676 /opt/gitlab/embedded/service/gitlab-rails/app/workers/object_storage/background_move_worker.rb:20:in `perform'
2018-10-25_02:33:41.58677 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.2.1/lib/sidekiq/processor.rb:185:in `execute_job'
2018-10-25_02:33:41.58677 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.2.1/lib/sidekiq/processor.rb:167:in `block (2 levels) in process'
2018-10-25_02:33:41.58678 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.2.1/lib/sidekiq/middleware/chain.rb:128:in `block in invoke'
2018-10-25_02:33:41.58678 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/sidekiq_status/server_middleware.rb:5:in `call'
2018-10-25_02:33:41.58679 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.2.1/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
2018-10-25_02:33:41.58679 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/sidekiq_middleware/request_store_middleware.rb:6:in `call'
2018-10-25_02:33:41.58679 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.2.1/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
2018-10-25_02:33:41.58679 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/sidekiq_middleware/shutdown.rb:52:in `call'
2018-10-25_02:33:41.58680 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.2.1/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
2018-10-25_02:33:41.58680 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/sidekiq_middleware.rb:13:in `block in call'
2018-10-25_02:33:41.58680 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/transaction.rb:53:in `run'
2018-10-25_02:33:41.58681 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/sidekiq_middleware.rb:13:in `call'
2018-10-25_02:33:41.58681 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.2.1/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
2018-10-25_02:33:41.58681 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.2.1/lib/sidekiq/middleware/server/active_record.rb:16:in `call'
2018-10-25_02:33:41.58681 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.2.1/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
2018-10-25_02:33:41.58682 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sentry-raven-2.7.2/lib/raven/integrations/sidekiq.rb:9:in `call'
2018-10-25_02:33:41.58682 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.2.1/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
2018-10-25_02:33:41.58682 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.2.1/lib/sidekiq/middleware/chain.rb:133:in `invoke'
2018-10-25_02:33:41.58682 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.2.1/lib/sidekiq/processor.rb:166:in `block in process'
2018-10-25_02:33:41.58682 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.2.1/lib/sidekiq/processor.rb:137:in `block (6 levels) in dispatch'
2018-10-25_02:33:41.58683 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.2.1/lib/sidekiq/job_retry.rb:98:in `local'
2018-10-25_02:33:41.58683 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.2.1/lib/sidekiq/processor.rb:136:in `block (5 levels) in dispatch'
2018-10-25_02:33:41.58684 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.2.1/lib/sidekiq.rb:36:in `block in <module:Sidekiq>'
2018-10-25_02:33:41.58684 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.2.1/lib/sidekiq/processor.rb:132:in `block (4 levels) in dispatch'
2018-10-25_02:33:41.58684 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.2.1/lib/sidekiq/processor.rb:217:in `stats'
2018-10-25_02:33:41.58684 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.2.1/lib/sidekiq/processor.rb:127:in `block (3 levels) in dispatch'
2018-10-25_02:33:41.58684 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.2.1/lib/sidekiq/job_logger.rb:8:in `call'
2018-10-25_02:33:41.58685 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.2.1/lib/sidekiq/processor.rb:126:in `block (2 levels) in dispatch'
2018-10-25_02:33:41.58685 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.2.1/lib/sidekiq/job_retry.rb:73:in `global'
2018-10-25_02:33:41.58686 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.2.1/lib/sidekiq/processor.rb:125:in `block in dispatch'
2018-10-25_02:33:41.58686 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.2.1/lib/sidekiq/logging.rb:48:in `with_context'
2018-10-25_02:33:41.58686 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.2.1/lib/sidekiq/logging.rb:42:in `with_job_hash_context'
2018-10-25_02:33:41.58686 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.2.1/lib/sidekiq/processor.rb:124:in `dispatch'
2018-10-25_02:33:41.58687 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.2.1/lib/sidekiq/processor.rb:165:in `process'
2018-10-25_02:33:41.58687 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.2.1/lib/sidekiq/processor.rb:83:in `process_one'
2018-10-25_02:33:41.58687 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.2.1/lib/sidekiq/processor.rb:71:in `run'
2018-10-25_02:33:41.58688 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.2.1/lib/sidekiq/util.rb:16:in `watchdog'
2018-10-25_02:33:41.58688 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.2.1/lib/sidekiq/util.rb:25:in `block in safe_thread'

Details of package version

Provide the package version installation details 
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                           Version              Architecture         Description
+++-==============================-====================-====================-=================================================================
un  gitlab-ce                                                    (no description available)
ii  gitlab-ee                      11.4.0-ee.0          amd64                GitLab Enterprise Edition (including NGINX, Postgres, Redis)

Environment details

  • Operating System: debian 9.5 amd64

  • Installation Target, remove incorrect values:

    • VM: Digital Ocean

  • Installation Type, remove incorrect values:

    • New Installation

  • Is there any other software running on the machine: no

  • Is this a single or multiple node installation? single

  • Resources

    • CPU: 2
    • Memory total: 4G

Configuration details

Provide the relevant sections of `/etc/gitlab/gitlab.rb` 
external_url 'https://xxx.xxx.xxx'
gitlab_rails['artifacts_object_store_enabled'] = true
gitlab_rails['artifacts_object_store_remote_directory'] = "example-bucket-name"
gitlab_rails['artifacts_object_store_connection'] = {
  'provider' => 'aliyun',
  'aliyun_accesskey_id' => '*********',
  'aliyun_accesskey_secret' => '**********',
  'aliyun_oss_bucket' => 'example-bucket-name',
  'aliyun_oss_location' => 'cn-shanghai',
  'aliyun_oss_endpoint' => 'oss-cn-shanghai.aliyuncs.com'
}