Geo: Fix repository verification on the primary
Summary
Primary repository verification is not working on latest nightly.
It looks like it would affect GitLab 12.9.0+.
Steps to reproduce
Create a project.
What is the current bug behavior?
Geo doesn't checksum it on the primary.
What is the expected correct behavior?
Geo checksums it on the primary.
Related logs
Sidekiq:
{"severity":"WARN","time":"2020-04-06T05:47:23.377Z","class":"Geo::RepositoryVerification::Primary::ShardWorker","retry":false,"queue":"geo:geo_repository_verification_primary_shard","queue_namespace":"geo","jid":"7dd206a1985d13519cfb3faa","created_at":"2020-04-06T05:47:23.337Z","meta.caller_id":"Geo::RepositoryVerification::Primary::BatchWorker","correlation_id":"0ba032d502e0639214eb00cd386432fa","enqueued_at":"2020-04-06T05:47:23.338Z","pid":2212,"message":"Geo::RepositoryVerification::Primary::ShardWorker JID-7dd206a1985d13519cfb3faa: fail: 0.032425 sec","job_status":"fail","scheduling_latency_s":0.006263,"duration":0.032425,"cpu_s":0.011594,"completed_at":"2020-04-06T05:47:23.377Z","error_message":"Geo secondary database is not configured","error_class":"Geo::TrackingBase::SecondaryNotConfigured","db_duration":1.382207999995444,"db_duration_s":0.001382207999995444}
{"severity":"WARN","time":"2020-04-06T05:47:23.378Z","error_class":"Geo::TrackingBase::SecondaryNotConfigured","error_message":"Geo secondary database is not configured","context":"Job raised exception","jobstr":"{\"class\":\"Geo::RepositoryVerification::Primary::ShardWorker\",\"args\":[\"default\"],\"retry\":false,\"queue\":\"geo:geo_repository_verification_primary_shard\",\"queue_namespace\":\"geo\",\"jid\":\"7dd206a1985d13519cfb3faa\",\"created_at\":1586152043.3374567,\"meta.caller_id\":\"Geo::RepositoryVerification::Primary::BatchWorker\",\"correlation_id\":\"0ba032d502e0639214eb00cd386432fa\",\"enqueued_at\":1586152043.3384297}","class":"Geo::RepositoryVerification::Primary::ShardWorker","args":["default"],"retry":false,"queue":"geo:geo_repository_verification_primary_shard","queue_namespace":"geo","jid":"7dd206a1985d13519cfb3faa","created_at":"2020-04-06T05:47:23.337Z","meta.caller_id":"Geo::RepositoryVerification::Primary::BatchWorker","correlation_id":"0ba032d502e0639214eb00cd386432fa","enqueued_at":"2020-04-06T05:47:23.338Z","error_backtrace":["ee/app/models/geo/tracking_base.rb:26:in `connection'","ee/lib/ee/gitlab/database.rb:31:in `geo_uncached_queries'","ee/app/workers/geo/scheduler/scheduler_worker.rb:173:in `update_pending_resources'","ee/app/workers/geo/scheduler/scheduler_worker.rb:47:in `block (2 levels) in perform'","ee/app/workers/geo/scheduler/scheduler_worker.rb:42:in `loop'","ee/app/workers/geo/scheduler/scheduler_worker.rb:42:in `block in perform'","app/services/concerns/exclusive_lease_guard.rb:29:in `try_obtain_lease'","ee/app/workers/geo/scheduler/scheduler_worker.rb:37:in `perform'","ee/app/workers/geo/scheduler/primary/scheduler_worker.rb:10:in `perform'","ee/app/workers/geo/repository_verification/primary/shard_worker.rb:16:in `perform'","lib/gitlab/with_request_store.rb:7:in `with_request_store'","lib/gitlab/sidekiq_daemon/monitor.rb:49:in `within_job'"]}
mkozono-ansible@mkozono-omnibus4838b-primary-geo:~$ sudo gitlab-rails console
--------------------------------------------------------------------------------
GitLab: 12.10.0-pre (35b18fe212e) EE
GitLab Shell: 12.0.0
PostgreSQL: 11.7
Geo enabled: yes
Geo server: primary
--------------------------------------------------------------------------------
Loading production environment (Rails 6.0.2)
irb(main):001:0> Gitlab.config.repositories.storages
=> {"default"=>#<Gitlab::GitalyClient::StorageSettings:0x00007f65ee60e7e8 @legacy_disk_path="/var/opt/gitlab/git-data/repositories", @hash={"path"=>Gitlab::GitalyClient::StorageSettings::Deprecated, "gitaly_address"=>"unix:/var/opt/gitlab/gitaly/gitaly.socket"}>}
irb(main):002:0> Geo::RepositoryVerification::Primary::ShardWorker.new.perform("default")
Traceback (most recent call last):
11: from (irb):2
10: from ee/app/workers/geo/repository_verification/primary/shard_worker.rb:16:in `perform'
9: from ee/app/workers/geo/scheduler/primary/scheduler_worker.rb:10:in `perform'
8: from ee/app/workers/geo/scheduler/scheduler_worker.rb:37:in `perform'
7: from app/services/concerns/exclusive_lease_guard.rb:29:in `try_obtain_lease'
6: from ee/app/workers/geo/scheduler/scheduler_worker.rb:42:in `block in perform'
5: from ee/app/workers/geo/scheduler/scheduler_worker.rb:42:in `loop'
4: from ee/app/workers/geo/scheduler/scheduler_worker.rb:47:in `block (2 levels) in perform'
3: from ee/app/workers/geo/scheduler/scheduler_worker.rb:173:in `update_pending_resources'
2: from ee/lib/ee/gitlab/database.rb:31:in `geo_uncached_queries'
1: from ee/app/models/geo/tracking_base.rb:26:in `connection'
Geo::TrackingBase::SecondaryNotConfigured (Geo secondary database is not configured)
Possible fixes
- We could use this block only on secondary schedulers https://gitlab.com/gitlab-org/gitlab/-/blob/v12.9.2-ee/ee/app/workers/geo/scheduler/scheduler_worker.rb#L173
- Or we could check for secondary https://gitlab.com/gitlab-org/gitlab/-/blob/v12.9.2-ee/ee/lib/ee/gitlab/database.rb#L31
Edited by Michael Kozono