ProjectSyncWorker crashing on the geo testbed
Geo::ProjectSyncWorker is failing crazy on the geo testbed.
https://log.gitlap.com/goto/030a1207ad5dccd3d1b118bed0396ba8
Blowing up here:
lib/gitlab/git/repository.rb
storage_path = Gitlab.config.repositories.storages[@storage]['path']
We recently trimmed down the number of storages so I'm guessing we are looking up paths for storages that don't exist:
irb(main):001:0> Gitlab.config.repositories.storages
=> {"default"=>{"path"=>"/var/opt/gitlab/git-data-file08/repositories", "gitaly_address"=>"tcp://10.213.2.108:9999"}, "nfs-file08"=>{"path"=>"/var/opt/gitlab/git-data-file08/repositories", "gitaly_address"=>"tcp://10.213.2.108:9999"}}
Full stack trace
2018-01-11_13:17:22.86588 sidekiq-besteffort-04 sidekiq-cluster: 2018-01-11T13:17:22.545Z 28456 TID-owqx2t1sk Geo::ProjectSyncWorker JID-cbfc2638a9e10cfe02fb0361 INFO: fail: 6.341 sec
2018-01-11_13:17:22.86595 sidekiq-besteffort-04 sidekiq-cluster: 2018-01-11T13:17:22.545Z 28456 TID-owqx2t1sk WARN: {"context":"Job raised exception","job":{"class":"Geo::ProjectSyncWorker","args":[4286134,"2018-01-11 12:54:25 +0000"],"retry":3,"queue":"geo_project_sync","dead":false,"jid":"cbfc2638a9e10cfe02fb0361","created_at":1515675265.9586377,"enqueued_at":1515675265.9777026},"jobstr":"{\"class\":\"Geo::ProjectSyncWorker\",\"args\":[4286134,\"2018-01-11 12:54:25 +0000\"],\"retry\":3,\"queue\":\"geo_project_sync\",\"dead\":false,\"jid\":\"cbfc2638a9e10cfe02fb0361\",\"created_at\":1515675265.9586377,\"enqueued_at\":1515675265.9777026}"}
2018-01-11_13:17:22.86651 sidekiq-besteffort-04 sidekiq-cluster: 2018-01-11T13:17:22.545Z 28456 TID-owqx2t1sk WARN: NoMethodError: undefined method `[]' for nil:NilClass
2018-01-11_13:17:22.86659 sidekiq-besteffort-04 sidekiq-cluster: 2018-01-11T13:17:22.545Z 28456 TID-owqx2t1sk WARN: /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/git/repository.rb:96:in `initialize'
2018-01-11_13:17:22.86666 sidekiq-besteffort-04 sidekiq-cluster: /opt/gitlab/embedded/service/gitlab-rails/app/models/repository.rb:1183:in `new'
2018-01-11_13:17:22.86673 sidekiq-besteffort-04 sidekiq-cluster: /opt/gitlab/embedded/service/gitlab-rails/app/models/repository.rb:1183:in `initialize_raw_repository'
2018-01-11_13:17:22.86680 sidekiq-besteffort-04 sidekiq-cluster: /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/instrumentation.rb:159:in `block in initialize_raw_repository'
2018-01-11_13:17:22.86688 sidekiq-besteffort-04 sidekiq-cluster: /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/method_call.rb:47:in `measure'
2018-01-11_13:17:22.86697 sidekiq-besteffort-04 sidekiq-cluster: /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/instrumentation.rb:159:in `initialize_raw_repository'
2018-01-11_13:17:22.86711 sidekiq-besteffort-04 sidekiq-cluster: /opt/gitlab/embedded/service/gitlab-rails/app/models/repository.rb:97:in `raw_repository'
2018-01-11_13:17:22.86717 sidekiq-besteffort-04 sidekiq-cluster: /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/instrumentation.rb:159:in `block in raw_repository'
2018-01-11_13:17:22.86722 sidekiq-besteffort-04 sidekiq-cluster: /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/method_call.rb:47:in `measure'
2018-01-11_13:17:22.86727 sidekiq-besteffort-04 sidekiq-cluster: /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/instrumentation.rb:159:in `raw_repository'
2018-01-11_13:17:22.86732 sidekiq-besteffort-04 sidekiq-cluster: /opt/gitlab/embedded/service/gitlab-rails/app/models/repository.rb:523:in `exists?'
2018-01-11_13:17:22.86738 sidekiq-besteffort-04 sidekiq-cluster: /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/instrumentation.rb:159:in `block in _uncached_exists?'
2018-01-11_13:17:22.86743 sidekiq-besteffort-04 sidekiq-cluster: /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/method_call.rb:47:in `measure'
2018-01-11_13:17:22.86747 sidekiq-besteffort-04 sidekiq-cluster: /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/instrumentation.rb:159:in `_uncached_exists?'
2018-01-11_13:17:22.86754 sidekiq-besteffort-04 sidekiq-cluster: /opt/gitlab/embedded/service/gitlab-rails/app/models/repository.rb:77:in `block (2 levels) in cache_method'
2018-01-11_13:17:22.86761 sidekiq-besteffort-04 sidekiq-cluster: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activesupport-4.2.10/lib/active_support/cache.rb:299:in `block in fetch'
2018-01-11_13:17:22.86766 sidekiq-besteffort-04 sidekiq-cluster: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activesupport-4.2.10/lib/active_support/cache.rb:585:in `block in save_block_result_to_cache'
2018-01-11_13:17:22.86773 sidekiq-besteffort-04 sidekiq-cluster: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activesupport-4.2.10/lib/active_support/cache.rb:547:in `block in instrument'
2018-01-11_13:17:22.86778 sidekiq-besteffort-04 sidekiq-cluster: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activesupport-4.2.10/lib/active_support/notifications.rb:164:in `block in instrument'
2018-01-11_13:17:22.86785 sidekiq-besteffort-04 sidekiq-cluster: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activesupport-4.2.10/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
2018-01-11_13:17:22.86795 sidekiq-besteffort-04 sidekiq-cluster: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activesupport-4.2.10/lib/active_support/notifications.rb:164:in `instrument'
2018-01-11_13:17:22.86802 sidekiq-besteffort-04 sidekiq-cluster: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activesupport-4.2.10/lib/active_support/cache.rb:547:in `instrument'
2018-01-11_13:17:22.86808 sidekiq-besteffort-04 sidekiq-cluster: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activesupport-4.2.10/lib/active_support/cache.rb:584:in `save_block_result_to_cache'
2018-01-11_13:17:22.86815 sidekiq-besteffort-04 sidekiq-cluster: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activesupport-4.2.10/lib/active_support/cache.rb:299:in `fetch'
2018-01-11_13:17:22.87196 sidekiq-besteffort-04 sidekiq-cluster: /opt/gitlab/embedded/service/gitlab-rails/lib/repository_cache.rb:20:in `fetch'
2018-01-11_13:17:22.87205 sidekiq-besteffort-04 sidekiq-cluster: /opt/gitlab/embedded/service/gitlab-rails/app/models/repository.rb:1070:in `cache_method_output'
2018-01-11_13:17:22.87211 sidekiq-besteffort-04 sidekiq-cluster: /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/instrumentation.rb:159:in `block in cache_method_output'
2018-01-11_13:17:22.87216 sidekiq-besteffort-04 sidekiq-cluster: /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/method_call.rb:47:in `measure'
2018-01-11_13:17:22.87221 sidekiq-besteffort-04 sidekiq-cluster: /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/instrumentation.rb:159:in `cache_method_output'
2018-01-11_13:17:22.87226 sidekiq-besteffort-04 sidekiq-cluster: /opt/gitlab/embedded/service/gitlab-rails/app/models/repository.rb:76:in `block in cache_method'
2018-01-11_13:17:22.87232 sidekiq-besteffort-04 sidekiq-cluster: /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/instrumentation.rb:159:in `block in exists?'
2018-01-11_13:17:22.87237 sidekiq-besteffort-04 sidekiq-cluster: /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/method_call.rb:47:in `measure'
2018-01-11_13:17:22.87242 sidekiq-besteffort-04 sidekiq-cluster: /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/instrumentation.rb:159:in `exists?'
2018-01-11_13:17:22.87247 sidekiq-besteffort-04 sidekiq-cluster: /opt/gitlab/embedded/service/gitlab-rails/app/models/project.rb:1197:in `repository_exists?'
2018-01-11_13:17:22.87252 sidekiq-besteffort-04 sidekiq-cluster: /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/instrumentation.rb:159:in `block in repository_exists?'
2018-01-11_13:17:22.87257 sidekiq-besteffort-04 sidekiq-cluster: /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/method_call.rb:47:in `measure'
2018-01-11_13:17:22.87262 sidekiq-besteffort-04 sidekiq-cluster: /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/instrumentation.rb:159:in `repository_exists?'
2018-01-11_13:17:22.87269 sidekiq-besteffort-04 sidekiq-cluster: /opt/gitlab/embedded/service/gitlab-rails/app/models/project.rb:1193:in `ensure_repository'
2018-01-11_13:17:22.87276 sidekiq-besteffort-04 sidekiq-cluster: /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/instrumentation.rb:159:in `block in ensure_repository'
2018-01-11_13:17:22.87283 sidekiq-besteffort-04 sidekiq-cluster: /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/method_call.rb:47:in `measure'
2018-01-11_13:17:22.87290 sidekiq-besteffort-04 sidekiq-cluster: /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/instrumentation.rb:159:in `ensure_repository'
2018-01-11_13:17:22.87298 sidekiq-besteffort-04 sidekiq-cluster: /opt/gitlab/embedded/service/gitlab-rails/ee/app/services/geo/repository_sync_service.rb:21:in `fetch_project_repository'
2018-01-11_13:17:22.87307 sidekiq-besteffort-04 sidekiq-cluster: /opt/gitlab/embedded/service/gitlab-rails/ee/app/services/geo/repository_sync_service.rb:8:in `sync_repository'
2018-01-11_13:17:22.87314 sidekiq-besteffort-04 sidekiq-cluster: /opt/gitlab/embedded/service/gitlab-rails/ee/app/services/geo/base_sync_service.rb:30:in `block in execute'
2018-01-11_13:17:22.87321 sidekiq-besteffort-04 sidekiq-cluster: /opt/gitlab/embedded/service/gitlab-rails/ee/app/services/concerns/exclusive_lease_guard.rb:22:in `try_obtain_lease'
2018-01-11_13:17:22.87329 sidekiq-besteffort-04 sidekiq-cluster: /opt/gitlab/embedded/service/gitlab-rails/ee/app/services/geo/base_sync_service.rb:26:in `execute'
2018-01-11_13:17:22.87340 sidekiq-besteffort-04 sidekiq-cluster: /opt/gitlab/embedded/service/gitlab-rails/ee/app/workers/geo/project_sync_worker.rb:22:in `perform'
2018-01-11_13:17:22.87347 sidekiq-besteffort-04 sidekiq-cluster: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:188:in `execute_job'
2018-01-11_13:17:22.87354 sidekiq-besteffort-04 sidekiq-cluster: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:170:in `block (2 levels) in process'
2018-01-11_13:17:22.87361 sidekiq-besteffort-04 sidekiq-cluster: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/middleware/chain.rb:128:in `block in invoke'
2018-01-11_13:17:22.87368 sidekiq-besteffort-04 sidekiq-cluster: /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/sidekiq_status/server_middleware.rb:5:in `call'
2018-01-11_13:17:22.87374 sidekiq-besteffort-04 sidekiq-cluster: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
2018-01-11_13:17:22.87381 sidekiq-besteffort-04 sidekiq-cluster: /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/sidekiq_middleware/request_store_middleware.rb:6:in `call'
2018-01-11_13:17:22.87388 sidekiq-besteffort-04 sidekiq-cluster: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
2018-01-11_13:17:22.87395 sidekiq-besteffort-04 sidekiq-cluster: /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/sidekiq_middleware/memory_killer.rb:16:in `call'
2018-01-11_13:17:22.87402 sidekiq-besteffort-04 sidekiq-cluster: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
2018-01-11_13:17:22.87411 sidekiq-besteffort-04 sidekiq-cluster: /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/sidekiq_middleware/arguments_logger.rb:6:in `call'
2018-01-11_13:17:22.87419 sidekiq-besteffort-04 sidekiq-cluster: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
2018-01-11_13:17:22.87425 sidekiq-besteffort-04 sidekiq-cluster: /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/sidekiq_middleware.rb:13:in `block in call'
2018-01-11_13:17:22.87432 sidekiq-besteffort-04 sidekiq-cluster: /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/transaction.rb:52:in `run'
2018-01-11_13:17:22.87439 sidekiq-besteffort-04 sidekiq-cluster: /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/sidekiq_middleware.rb:13:in `call'
2018-01-11_13:17:22.87446 sidekiq-besteffort-04 sidekiq-cluster: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
2018-01-11_13:17:22.87456 sidekiq-besteffort-04 sidekiq-cluster: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/middleware/server/active_record.rb:15:in `call'
2018-01-11_13:17:22.87463 sidekiq-besteffort-04 sidekiq-cluster: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
2018-01-11_13:17:22.87471 sidekiq-besteffort-04 sidekiq-cluster: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sentry-raven-2.5.3/lib/raven/integrations/sidekiq.rb:7:in `call'
2018-01-11_13:17:22.87478 sidekiq-besteffort-04 sidekiq-cluster: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
2018-01-11_13:17:22.87486 sidekiq-besteffort-04 sidekiq-cluster: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/middleware/chain.rb:133:in `invoke'
2018-01-11_13:17:22.87492 sidekiq-besteffort-04 sidekiq-cluster: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:169:in `block in process'
2018-01-11_13:17:22.87499 sidekiq-besteffort-04 sidekiq-cluster: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:141:in `block (6 levels) in dispatch'
2018-01-11_13:17:22.87508 sidekiq-besteffort-04 sidekiq-cluster: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/job_retry.rb:97:in `local'
2018-01-11_13:17:22.87515 sidekiq-besteffort-04 sidekiq-cluster: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:140:in `block (5 levels) in dispatch'
2018-01-11_13:17:22.87522 sidekiq-besteffort-04 sidekiq-cluster: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq.rb:36:in `block in <module:Sidekiq>'
2018-01-11_13:17:22.87528 sidekiq-besteffort-04 sidekiq-cluster: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:136:in `block (4 levels) in dispatch'
2018-01-11_13:17:22.87536 sidekiq-besteffort-04 sidekiq-cluster: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:204:in `stats'
2018-01-11_13:17:22.87543 sidekiq-besteffort-04 sidekiq-cluster: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:131:in `block (3 levels) in dispatch'
2018-01-11_13:17:22.87549 sidekiq-besteffort-04 sidekiq-cluster: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/job_logger.rb:7:in `call'
2018-01-11_13:17:22.87556 sidekiq-besteffort-04 sidekiq-cluster: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:130:in `block (2 levels) in dispatch'
2018-01-11_13:17:22.87563 sidekiq-besteffort-04 sidekiq-cluster: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/job_retry.rb:72:in `global'
2018-01-11_13:17:22.87569 sidekiq-besteffort-04 sidekiq-cluster: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:129:in `block in dispatch'
2018-01-11_13:17:22.87576 sidekiq-besteffort-04 sidekiq-cluster: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/logging.rb:44:in `with_context'
2018-01-11_13:17:22.87583 sidekiq-besteffort-04 sidekiq-cluster: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/logging.rb:38:in `with_job_hash_context'
2018-01-11_13:17:22.87589 sidekiq-besteffort-04 sidekiq-cluster: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:128:in `dispatch'
2018-01-11_13:17:22.87597 sidekiq-besteffort-04 sidekiq-cluster: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:168:in `process'
2018-01-11_13:17:22.87603 sidekiq-besteffort-04 sidekiq-cluster: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:85:in `process_one'
2018-01-11_13:17:22.87610 sidekiq-besteffort-04 sidekiq-cluster: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:73:in `run'
2018-01-11_13:17:22.87617 sidekiq-besteffort-04 sidekiq-cluster: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/util.rb:16:in `watchdog'
2018-01-11_13:17:22.87624 sidekiq-besteffort-04 sidekiq-cluster: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/util.rb:25:in `block in safe_thread'