Background upload of LFS does not work with google_json API
Summary
Background uploading of LFS objects when using the Google Storage JSON API fails.
Steps to reproduce
Set up Google Storage for LFS using the JSON API
What is the current bug behavior?
Sidekiq fails to upload the objects
What is the expected correct behavior?
Sidekiq would upload the objects.
Relevant logs and/or screenshots
2018-03-13_16:37:17.90189 2018-03-13T16:37:17.901Z 2041 TID-ox0hiaf58 ObjectStorage::BackgroundMoveWorker JID-a15c34cd1df2f388f51577ab INFO: start
2018-03-13_16:37:17.93175 2018-03-13T16:37:17.931Z 2041 TID-ox0hiaf58 ObjectStorage::BackgroundMoveWorker JID-a15c34cd1df2f388f51577ab INFO: fail: 0.03 sec
2018-03-13_16:37:17.93249 2018-03-13T16:37:17.932Z 2041 TID-ox0hiaf58 WARN: {"context":"Job raised exception","job":{"class":"ObjectStorage::BackgroundMoveWorker","args":["LfsObjectUploader","LfsObject","
file",554],"retry":5,"queue":"object_storage:object_storage_background_move","queue_namespace":"object_storage","jid":"a15c34cd1df2f388f51577ab","created_at":1520958711.5200455,"enqueued_at":1520959037.89
77094,"error_message":"string contains null byte","error_class":"ArgumentError","failed_at":1520958712.6831982,"retry_count":3,"retried_at":1520958935.3825207},"jobstr":"{\"class\":\"ObjectStorage::Backgr
oundMoveWorker\",\"args\":[\"LfsObjectUploader\",\"LfsObject\",\"file\",554],\"retry\":5,\"queue\":\"object_storage:object_storage_background_move\",\"queue_namespace\":\"object_storage\",\"jid\":\"a15c34
cd1df2f388f51577ab\",\"created_at\":1520958711.5200455,\"enqueued_at\":1520959037.8977094,\"error_message\":\"string contains null byte\",\"error_class\":\"ArgumentError\",\"failed_at\":1520958712.6831982
,\"retry_count\":3,\"retried_at\":1520958935.3825207}"}
2018-03-13_16:37:17.93256 2018-03-13T16:37:17.932Z 2041 TID-ox0hiaf58 WARN: ArgumentError: string contains null byte
2018-03-13_16:37:17.93272 2018-03-13T16:37:17.932Z 2041 TID-ox0hiaf58 WARN: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/google-api-client-0.19.8/lib/google/apis/core/upload.rb:56:in `initialize'
2018-03-13_16:37:17.93274 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/google-api-client-0.19.8/lib/google/apis/core/upload.rb:56:in `new'
2018-03-13_16:37:17.93274 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/google-api-client-0.19.8/lib/google/apis/core/upload.rb:56:in `prepare!'
2018-03-13_16:37:17.93275 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/google-api-client-0.19.8/lib/google/apis/core/upload.rb:141:in `prepare!'
2018-03-13_16:37:17.93276 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/google-api-client-0.19.8/lib/google/apis/core/http_command.rb:91:in `execute'
2018-03-13_16:37:17.93276 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/google-api-client-0.19.8/lib/google/apis/core/base_service.rb:360:in `execute_or_queue_command'
2018-03-13_16:37:17.93277 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/google-api-client-0.19.8/generated/google/apis/storage_v1/service.rb:1981:in `insert_object'
2018-03-13_16:37:17.93278 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/fog-google-1.3.0/lib/fog/storage/google_json/requests/put_object.rb:68:in `put_object'
2018-03-13_16:37:17.93280 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/fog-google-1.3.0/lib/fog/storage/google_json/models/file.rb:78:in `save'
2018-03-13_16:37:17.93281 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/fog-core-1.45.0/lib/fog/core/collection.rb:50:in `create'
2018-03-13_16:37:17.93281 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/carrierwave-1.2.1/lib/carrierwave/storage/fog.rb:315:in `store'
2018-03-13_16:37:17.93282 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/carrierwave-1.2.1/lib/carrierwave/storage/fog.rb:78:in `store!'
2018-03-13_16:37:17.93283 /opt/gitlab/embedded/service/gitlab-rails/ee/app/uploaders/object_storage.rb:416:in `block (2 levels) in unsafe_migrate!'
2018-03-13_16:37:17.93283 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/carrierwave-1.2.1/lib/carrierwave/uploader/callbacks.rb:15:in `with_callbacks'
2018-03-13_16:37:17.93284 /opt/gitlab/embedded/service/gitlab-rails/ee/app/uploaders/object_storage.rb:415:in `block in unsafe_migrate!'
2018-03-13_16:37:17.93285 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/carrierwave-1.2.1/lib/carrierwave/uploader/callbacks.rb:15:in `with_callbacks'
2018-03-13_16:37:17.93286 /opt/gitlab/embedded/service/gitlab-rails/ee/app/uploaders/object_storage.rb:414:in `unsafe_migrate!'
2018-03-13_16:37:17.93288 /opt/gitlab/embedded/service/gitlab-rails/ee/app/uploaders/object_storage.rb:257:in `migrate!'
2018-03-13_16:37:17.93288 /opt/gitlab/embedded/service/gitlab-rails/ee/app/workers/object_storage/background_move_worker.rb:19:in `perform'
2018-03-13_16:37:17.93289 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:188:in `execute_job'
2018-03-13_16:37:17.93290 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:170:in `block (2 levels) in process'
2018-03-13_16:37:17.93290 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/middleware/chain.rb:128:in `block in invoke'
2018-03-13_16:37:17.93291 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/sidekiq_status/server_middleware.rb:5:in `call'
2018-03-13_16:37:17.93292 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
2018-03-13_16:37:17.93292 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/sidekiq_middleware/request_store_middleware.rb:6:in `call'
2018-03-13_16:37:17.93293 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
2018-03-13_16:37:17.93295 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/sidekiq_middleware/shutdown.rb:52:in `call'
2018-03-13_16:37:17.93297 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
2018-03-13_16:37:17.93298 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/sidekiq_middleware.rb:13:in `block in call'
2018-03-13_16:37:17.93299 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/transaction.rb:53:in `run'
2018-03-13_16:37:17.93300 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/sidekiq_middleware.rb:13:in `call'
2018-03-13_16:37:17.93300 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
2018-03-13_16:37:17.93301 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/middleware/server/active_record.rb:15:in `call'
2018-03-13_16:37:17.93302 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
2018-03-13_16:37:17.93302 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sentry-raven-2.5.3/lib/raven/integrations/sidekiq.rb:7:in `call'
2018-03-13_16:37:17.93303 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
2018-03-13_16:37:17.93305 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/middleware/chain.rb:133:in `invoke'
2018-03-13_16:37:17.93306 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:169:in `block in process'
2018-03-13_16:37:17.93307 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:141:in `block (6 levels) in dispatch'
2018-03-13_16:37:17.93307 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/job_retry.rb:97:in `local'
2018-03-13_16:37:17.93308 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:140:in `block (5 levels) in dispatch'
2018-03-13_16:37:17.93309 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq.rb:36:in `block in <module:Sidekiq>'
2018-03-13_16:37:17.93309 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:136:in `block (4 levels) in dispatch'
2018-03-13_16:37:17.93310 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:204:in `stats'
2018-03-13_16:37:17.93311 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:131:in `block (3 levels) in dispatch'
2018-03-13_16:37:17.93313 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/job_logger.rb:7:in `call'
2018-03-13_16:37:17.93314 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:130:in `block (2 levels) in dispatch'
2018-03-13_16:37:17.93314 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/job_retry.rb:72:in `global'
2018-03-13_16:37:17.93315 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:129:in `block in dispatch'
2018-03-13_16:37:17.93316 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/logging.rb:44:in `with_context'
2018-03-13_16:37:17.93316 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/logging.rb:38:in `with_job_hash_context'
2018-03-13_16:37:17.93317 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:128:in `dispatch'
2018-03-13_16:37:17.93318 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:168:in `process'
2018-03-13_16:37:17.93318 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:85:in `process_one'
2018-03-13_16:37:17.93321 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:73:in `run'
2018-03-13_16:37:17.93321 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/util.rb:16:in `watchdog'
2018-03-13_16:37:17.93322 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/util.rb:25:in `block in safe_thread'