can not use aliyun object storage for job artifacts
Summary
use aliyun oss object storage for job artifacts, when i migrate artifacts, it can not work
Steps to reproduce
you should have an aliyun oss bucket, and right access policy setup, you need the access key id and secret.
gitlab.rb
gitlab_rails['artifacts_object_store_enabled'] = true
gitlab_rails['artifacts_object_store_remote_directory'] = "example-bucket-name"
gitlab_rails['artifacts_object_store_connection'] = {
'provider' => 'aliyun',
'aliyun_accesskey_id' => '*********',
'aliyun_accesskey_secret' => '**********',
'aliyun_oss_bucket' => 'example-bucket-name',
'aliyun_oss_location' => 'cn-shanghai',
'aliyun_oss_endpoint' => 'oss-cn-shanghai.aliyuncs.com'
}
then run follow command
$ sudo gitlab-ctl reconfigure
$ sudo gitlab-rake gitlab:artifacts:migrate
What is the current bug behavior?
it return error logs
I, [2018-10-25T02:33:53.885875 #8127] INFO -- : Starting transfer of artifacts
E, [2018-10-25T02:33:54.027448 #8127] ERROR -- : Failed to transfer artifacts of 1 with error: no implicit conversion of nil into String
What is the expected correct behavior?
it should exit with success, and migrate local artifacts to object storage
Relevant logs
==> /var/log/gitlab/sidekiq/current <==
2018-10-25_02:33:41.48735 2018-10-25T02:33:41.487Z 7886 TID-otl43zj7u ObjectStorage::BackgroundMoveWorker JID-52b56092c457b8d21b14afa8 INFO: start
2018-10-25_02:33:41.58575 2018-10-25T02:33:41.585Z 7886 TID-otl43zj7u ObjectStorage::BackgroundMoveWorker JID-52b56092c457b8d21b14afa8 INFO: Adding dead ObjectStorage::BackgroundMoveWorker job 52b56092c457b8d21b14afa8
2018-10-25_02:33:41.58645 2018-10-25T02:33:41.586Z 7886 TID-otl43zj7u ObjectStorage::BackgroundMoveWorker JID-52b56092c457b8d21b14afa8 INFO: fail: 0.099 sec
2018-10-25_02:33:41.58657 2018-10-25T02:33:41.586Z 7886 TID-otl43zj7u WARN: {"context":"Job raised exception","job":{"class":"ObjectStorage::BackgroundMoveWorker","args":["JobArtifactUploader","Ci::JobArtifact","file",2],"retry":5,"queue":"object_storage:object_storage_background_move","queue_namespace":"object_storage","jid":"52b56092c457b8d21b14afa8","created_at":1540434104.7881083,"enqueued_at":1540434821.4865055,"error_message":"no implicit conversion of nil into String","error_class":"TypeError","failed_at":1540434104.9099479,"retry_count":4,"retried_at":1540434467.7912023},"jobstr":"{\"class\":\"ObjectStorage::BackgroundMoveWorker\",\"args\":[\"JobArtifactUploader\",\"Ci::JobArtifact\",\"file\",2],\"retry\":5,\"queue\":\"object_storage:object_storage_background_move\",\"queue_namespace\":\"object_storage\",\"jid\":\"52b56092c457b8d21b14afa8\",\"created_at\":1540434104.7881083,\"enqueued_at\":1540434821.4865055,\"error_message\":\"no implicit conversion of nil into String\",\"error_class\":\"TypeError\",\"failed_at\":1540434104.9099479,\"retry_count\":4,\"retried_at\":1540434467.7912023}"}
2018-10-25_02:33:41.58663 2018-10-25T02:33:41.586Z 7886 TID-otl43zj7u WARN: TypeError: no implicit conversion of nil into String
2018-10-25_02:33:41.58671 2018-10-25T02:33:41.586Z 7886 TID-otl43zj7u WARN: /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/fog-aliyun-0.2.0/lib/fog/aliyun/storage.rb:113:in `+'
2018-10-25_02:33:41.58671 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/fog-aliyun-0.2.0/lib/fog/aliyun/storage.rb:113:in `request'
2018-10-25_02:33:41.58672 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/fog-aliyun-0.2.0/lib/fog/aliyun/requests/storage/get_bucket.rb:23:in `get_bucket_location'
2018-10-25_02:33:41.58672 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/fog-aliyun-0.2.0/lib/fog/aliyun/requests/storage/put_object.rb:13:in `put_object'
2018-10-25_02:33:41.58673 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/fog-aliyun-0.2.0/lib/fog/aliyun/models/storage/file.rb:113:in `save'
2018-10-25_02:33:41.58673 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/fog-core-1.45.0/lib/fog/core/collection.rb:50:in `create'
2018-10-25_02:33:41.58674 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/carrierwave-1.2.3/lib/carrierwave/storage/fog.rb:325:in `store'
2018-10-25_02:33:41.58674 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/carrierwave-1.2.3/lib/carrierwave/storage/fog.rb:78:in `store!'
2018-10-25_02:33:41.58674 /opt/gitlab/embedded/service/gitlab-rails/app/uploaders/object_storage.rb:430:in `block (2 levels) in unsafe_migrate!'
2018-10-25_02:33:41.58675 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/carrierwave-1.2.3/lib/carrierwave/uploader/callbacks.rb:15:in `with_callbacks'
2018-10-25_02:33:41.58675 /opt/gitlab/embedded/service/gitlab-rails/app/uploaders/object_storage.rb:429:in `block in unsafe_migrate!'
2018-10-25_02:33:41.58675 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/carrierwave-1.2.3/lib/carrierwave/uploader/callbacks.rb:15:in `with_callbacks'
2018-10-25_02:33:41.58676 /opt/gitlab/embedded/service/gitlab-rails/app/uploaders/object_storage.rb:428:in `unsafe_migrate!'
2018-10-25_02:33:41.58676 /opt/gitlab/embedded/service/gitlab-rails/app/uploaders/object_storage.rb:262:in `block in migrate!'
2018-10-25_02:33:41.58676 /opt/gitlab/embedded/service/gitlab-rails/app/uploaders/object_storage.rb:407:in `with_exclusive_lease'
2018-10-25_02:33:41.58676 /opt/gitlab/embedded/service/gitlab-rails/app/uploaders/object_storage.rb:261:in `migrate!'
2018-10-25_02:33:41.58676 /opt/gitlab/embedded/service/gitlab-rails/app/workers/object_storage/background_move_worker.rb:20:in `perform'
2018-10-25_02:33:41.58677 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.2.1/lib/sidekiq/processor.rb:185:in `execute_job'
2018-10-25_02:33:41.58677 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.2.1/lib/sidekiq/processor.rb:167:in `block (2 levels) in process'
2018-10-25_02:33:41.58678 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.2.1/lib/sidekiq/middleware/chain.rb:128:in `block in invoke'
2018-10-25_02:33:41.58678 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/sidekiq_status/server_middleware.rb:5:in `call'
2018-10-25_02:33:41.58679 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.2.1/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
2018-10-25_02:33:41.58679 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/sidekiq_middleware/request_store_middleware.rb:6:in `call'
2018-10-25_02:33:41.58679 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.2.1/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
2018-10-25_02:33:41.58679 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/sidekiq_middleware/shutdown.rb:52:in `call'
2018-10-25_02:33:41.58680 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.2.1/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
2018-10-25_02:33:41.58680 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/sidekiq_middleware.rb:13:in `block in call'
2018-10-25_02:33:41.58680 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/transaction.rb:53:in `run'
2018-10-25_02:33:41.58681 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/sidekiq_middleware.rb:13:in `call'
2018-10-25_02:33:41.58681 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.2.1/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
2018-10-25_02:33:41.58681 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.2.1/lib/sidekiq/middleware/server/active_record.rb:16:in `call'
2018-10-25_02:33:41.58681 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.2.1/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
2018-10-25_02:33:41.58682 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sentry-raven-2.7.2/lib/raven/integrations/sidekiq.rb:9:in `call'
2018-10-25_02:33:41.58682 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.2.1/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
2018-10-25_02:33:41.58682 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.2.1/lib/sidekiq/middleware/chain.rb:133:in `invoke'
2018-10-25_02:33:41.58682 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.2.1/lib/sidekiq/processor.rb:166:in `block in process'
2018-10-25_02:33:41.58682 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.2.1/lib/sidekiq/processor.rb:137:in `block (6 levels) in dispatch'
2018-10-25_02:33:41.58683 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.2.1/lib/sidekiq/job_retry.rb:98:in `local'
2018-10-25_02:33:41.58683 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.2.1/lib/sidekiq/processor.rb:136:in `block (5 levels) in dispatch'
2018-10-25_02:33:41.58684 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.2.1/lib/sidekiq.rb:36:in `block in <module:Sidekiq>'
2018-10-25_02:33:41.58684 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.2.1/lib/sidekiq/processor.rb:132:in `block (4 levels) in dispatch'
2018-10-25_02:33:41.58684 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.2.1/lib/sidekiq/processor.rb:217:in `stats'
2018-10-25_02:33:41.58684 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.2.1/lib/sidekiq/processor.rb:127:in `block (3 levels) in dispatch'
2018-10-25_02:33:41.58684 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.2.1/lib/sidekiq/job_logger.rb:8:in `call'
2018-10-25_02:33:41.58685 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.2.1/lib/sidekiq/processor.rb:126:in `block (2 levels) in dispatch'
2018-10-25_02:33:41.58685 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.2.1/lib/sidekiq/job_retry.rb:73:in `global'
2018-10-25_02:33:41.58686 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.2.1/lib/sidekiq/processor.rb:125:in `block in dispatch'
2018-10-25_02:33:41.58686 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.2.1/lib/sidekiq/logging.rb:48:in `with_context'
2018-10-25_02:33:41.58686 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.2.1/lib/sidekiq/logging.rb:42:in `with_job_hash_context'
2018-10-25_02:33:41.58686 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.2.1/lib/sidekiq/processor.rb:124:in `dispatch'
2018-10-25_02:33:41.58687 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.2.1/lib/sidekiq/processor.rb:165:in `process'
2018-10-25_02:33:41.58687 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.2.1/lib/sidekiq/processor.rb:83:in `process_one'
2018-10-25_02:33:41.58687 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.2.1/lib/sidekiq/processor.rb:71:in `run'
2018-10-25_02:33:41.58688 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.2.1/lib/sidekiq/util.rb:16:in `watchdog'
2018-10-25_02:33:41.58688 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.2.1/lib/sidekiq/util.rb:25:in `block in safe_thread'
Details of package version
Provide the package version installation details
Desired=Unknown/Install/Remove/Purge/Hold | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad) ||/ Name Version Architecture Description +++-==============================-====================-====================-================================================================= un gitlab-ce (no description available) ii gitlab-ee 11.4.0-ee.0 amd64 GitLab Enterprise Edition (including NGINX, Postgres, Redis)
Environment details
-
Operating System:
debian 9.5 amd64
-
Installation Target, remove incorrect values:
- VM: Digital Ocean
-
Installation Type, remove incorrect values:
- New Installation
-
Is there any other software running on the machine: no
-
Is this a single or multiple node installation? single
-
Resources
- CPU:
2
- Memory total:
4G
- CPU:
Configuration details
Provide the relevant sections of `/etc/gitlab/gitlab.rb`
external_url 'https://xxx.xxx.xxx' gitlab_rails['artifacts_object_store_enabled'] = true gitlab_rails['artifacts_object_store_remote_directory'] = "example-bucket-name" gitlab_rails['artifacts_object_store_connection'] = { 'provider' => 'aliyun', 'aliyun_accesskey_id' => '*********', 'aliyun_accesskey_secret' => '**********', 'aliyun_oss_bucket' => 'example-bucket-name', 'aliyun_oss_location' => 'cn-shanghai', 'aliyun_oss_endpoint' => 'oss-cn-shanghai.aliyuncs.com' }